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Abstract 


Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  interested  in  researching  training 
techniques  to  prepare  Canadian  Forces  (CF)  commanders  and  staff  for  decision  making  in  complex 
and  dynamic  environments.  The  complex  and  dynamic  nature  of  various  types  of  operations  (e.g., 
effects-based  operations,  stability  operations,  counter-IED,  counter-insurgency)  pose  specific 
cognitive  challenges  on  the  decision-making  process  that  the  current  training  regimen  of  military 
commanders  does  not  directly  address.  This  project  serves  to  provide  understanding  of  underlying 
cognitive  mechanisms  within  DDM  as  a  contribution  to  determining  how  best  to  improve  the  DDM 
process  and  training.  A  brief  review  of  four  recognition  processes,  namely  pattern  recognition,  cue 
integration,  heuristics,  and  biases,  was  conducted  as  Phase  I  of  a  two  phase  process.  Based  on  this 
review,  cue  integration  was  selected  as  an  area  of  key  interest  and  was  the  focus  of  a  more 
comprehensive  review  (Phase  II).  Results  from  this  review  indicated  that  the  cue  integration 
literature  within  DDM  is  sparse.  A  significant  amount  literature  concerning  more  simplistic 
decision  making  thus  was  used  to  infer  the  role  of  cue  integration  within  DDM.  Particular  aspects 
of  cue  integration,  namely  perception,  working  memory,  long  term  memory,  and  feedback  were 
considered  to  play  an  important  role  in  DDM.  Training  implications  including  recommendations  of 
what  and  how  to  train  are  considered.  Concluding  remarks  and  future  recommendations  are  also 
discussed 
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Resume 


Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  Toronto  s’interesse  a  la  recherche 
sur  les  techniques  de  formation  destinees  a  preparer  les  commandants  et  le  personnel  des  etats- 
majors  des  Forces  canadiennes  (FC)  a  prendre  des  decisions  dans  des  environnements  complexes  et 
dynamiques.  La  nature  complexe  et  dynamique  de  differents  types  d’operations  (p.  ex.,  les 
operations  basees  sur  les  effets,  les  operations  de  stability  la  lutte  aux  IED,  les  mesures  anti- 
insurrectionnelles)  pose  des  defis  particuliers  au  niveau  cognitif  dans  le  processus  de  prise  de 
decisions  qui  ne  sont  pas  abordes  directement  dans  le  cadre  de  l’actuel  regime  destruction  des 
commandants  militaires.  Le  present  projet  sert  a  comprendre  les  mecanismes  cognitifs  sous-jacents 
de  la  prise  de  decisions  dynamique  (PDD)  en  tant  qu’elements  contribuant  a  determiner  quelle  est 
la  meilleure  fa$on  d’ameliorer  le  processus  de  PDD  et  la  formation.  La  premiere  des  deux  phases 
de  ce  processus  etait  une  etude  breve  de  quatre  processus  de  reconnaissance,  a  savoir  la 
reconnaissance  des  tendances,  Pintegration  des  indices,  les  heuristiques  et  les  biais.  En  fonction  des 
resultats  de  cette  etude,  Pintegration  des  indices  a  ete  choisie  comme  domaine  d'interet  cle  et  a  fait 
l'objet  d'une  etude  plus  detaillee  dans  la  Phase  II.  Les  resultats  de  cette  etude  indiquaient  qu’il  y 
avait  peu  de  documentation  sur  Pintegration  des  indices  dans  le  domaine  de  la  PDD.  Une  quantite 
importante  de  documentation  relative  aux  contextes  de  prise  de  decisions  plus  simples  a  done  ainsi 
ete  utilisee  pour  induire  le  role  de  Pintegration  des  indices  dans  la  PDD.  On  estime  que  des  aspects 
particuliers  de  Pintegration  des  indices,  e'est-a-dire  la  perception,  la  memoire  de  travail,  la  memoire 
a  long  terme  et  la  retroaction,  jouent  un  role  important  dans  la  PDD.  Les  implications  au  niveau  de 
la  formation  sont  abordees.  Des  recommandations  sont  examinees  sur  ce  qui  devrait  faire  l'objet  de 
la  formation  et  la  fa£on  de  former  les  gens.  Le  present  document  se  termine  par  un  commentaire  de 
conclusion  et  des  recommandations  sur  les  recherches  a  venir. 
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Executive  Summary 

Cue  Integration  in  Dynamic  Decision  Making 

Lisa  Rehak,  Tamsen  Taylor,  Cheryl  Karthaus  &  Lora  Bruyn  Martin; 

Human  systems®  Incorporated 

DRDC  Toronto  No.  CR201 0-047;  Defence  Research  and  Development  Canada 
-  Toronto;  July  2010. 

Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  interested  in  researching  training 
techniques  to  prepare  Canadian  Forces  (CF)  commanders  and  staff  for  decision  making  in  complex 
and  dynamic  environments.  The  complex  and  dynamic  nature  of  various  types  of  operations  (e.g., 
effects-based  operations,  stability  operations,  counter-IED,  counter-insurgency)  pose  specific 
cognitive  challenges  on  the  decision-making  process  that  the  current  training  regimen  of  military 
commanders  does  not  directly  address. 

In  the  psychology  literature,  decision  making  in  complex  and  dynamic  environments  is  generally 
referred  to  as  dynamic  decision  making  (DDM).  Unique  challenges  associated  with  DDM  are  the 
uncertainties  within  the  situation  (e.g.,  ambiguous  information),  time  constraints,  high  levels  of 
risk,  ill  structured  problems,  high  levels  of  complexity,  situation  changes,  and  delays  in  feedback. 

In  working  through  such  intricate  problems  there  is  a  significant  need  to  take  multiple  variables 
into  account  and  to  integrate  numerous  pieces  of  information  gathered  over  time,  in  order  to  make 
appropriate  decisions  in  a  timely  manner.  Accordingly,  much  of  the  research  conducted  on  DDM 
assumes  that  recognition  processes  are  crucial  to  successful  DDM  (see,  e.g.,  Brehmer,  1990; 
Gonzalez,  Lerch,  &  Lebiere,  2003).  Four  recognition  related  processes  include  pattern  recognition, 
cue  integration,  heuristics,  and  biases. 

In  efforts  to  understand  particular  challenges  within  these  processes  this  project  examines  the 
psychological  literature  of  underlying  cognitive  mechanisms  of  DDM.  Specifically,  objectives  of 
this  review  were  to  identify  and  review  relevant  literature  on  role  of  pattern  recognition,  cue 
integration,  heuristics,  and  biases  in  relation  to  DDM,  in  particular  to  identify  challenges  to  and 
failures  of  normative  reasoning;  to  review  more  thoroughly  the  literature  on  one  particular  area;  to 
construct  a  (or  build  upon  an  existing)  conceptual  framework  for  relating  the  selected  area  to 
DDM,  identifying  gaps  in  the  literature,  guiding  areas  for  future  research,  and  identifying 
constraints  and  implications  for  DDM  training. 

This  project  was  conducted  in  two  phases.  Phase  I  consists  of  a  brief  review  of  the  literature  on 
pattern  recognition,  cue  integration,  heuristics,  and  biases,  as  they  relate  to  DDM.  A  total  of  60 
articles  were  initially  reviewed  to  satisfy  the  requirements  of  Phase  I.  Within  each  of  these  four 
areas  general  themes,  challenges,  and  training  implications  are  discussed. 

Through  a  brief  analysis,  cue  integration  was  selected  as  the  area  of  focus  in  Phase  II.  In  this  phase 
an  additional  17  articles  specific  to  cue  integration  were  reviewed.  Findings  indicated  that  the  cue 
integration  literature  within  DDM  is  sparse.  A  significant  amount  literature  concerning  more 
simplistic  decision  making  thus  was  used  to  infer  the  role  of  cue  integration  within  DDM. 

Particular  aspects  of  cue  integration,  namely  perception,  working  memory,  long  term  memory,  and 
feedback  were  considered  to  play  an  important  role  in  DDM.  Training  implications  including 
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recommendations  of  what  and  how  to  train  are  considered.  Concluding  remarks  and  future 
recommendations  are  also  discussed  with  a  specific  focus  on  the  CF. 

Overall,  this  project  serves  to  provide  understanding  of  underlying  cognitive  mechanisms  within 
the  DDM  as  a  contribution  to  determining  how  best  to  improve  the  DDM  process  and  training  for 
the  future  CF. 
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Integration  des  indices  dans  la  prise  de  decision 
dynamique 

Lisa  Rehak,  Tamsen  Taylor,  Cheryl  Karthaus  &  Lora  Bruyn  Martin; 

Human  systems®  Incorporated 

RDDC  Toronto  No.  CR201 0-047;  R&D  pour  la  defense  Canada  -  Toronto; 
juillet  2010. 

Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  Toronto  s’interesse  a  la  recherche 
sur  les  techniques  de  formation  destinees  a  preparer  les  commandants  et  Ie  personnel  des  etats- 
majors  des  Forces  canadiennes  (FC)  a  prendre  des  decisions  dans  des  environnements  complexes  et 
dynamiques.  La  nature  complexe  et  dynamique  de  differents  types  d’operations  (p.  ex.,  les 
operations  basees  sur  les  effets,  les  operations  de  stabilite,  la  Iutte  aux  IED  et  les  mesures  anti- 
insurrectionnelles)  pose  des  defis  particuliers  au  niveau  cognitif  dans  le  processus  de  prise  de 
decisions  qui  ne  sont  pas  abordes  directement  dans  le  cadre  de  Pactuel  regime  d’instruction  des 
commandants  militaires. 

Dans  les  ecrits  en  psychologie,  la  prise  de  decisions  dans  les  environnements  complexes  et 
dynamiques  est  habituellement  designee  sous  le  nom  de  prise  de  decisions  dynamique  (PDD).  Les 
defis  propres  a  la  PDD  sont  les  incertitudes  presentes  dans  la  situation  (p.  ex.,  des  renseignements 
ambigus),  les  contraintes  de  temps,  les  hauts  niveaux  de  risques,  la  mauvaise  structuration  des 
problemes,  les  hauts  niveaux  de  complexity  les  changements  de  situation  et  les  retards  de  la 
presentation  de  retroaction.  Lorsqu'on  travaille  avec  des  problemes  si  compliques,  il  est  necessaire 
de  tenir  compte  de  nombreuses  variables  et  d'integrer  un  grand  nombre  d'elements  d'information 
recueillis  sur  une  certaine  periode  de  temps  afin  de  prendre  les  decisions  appropriees  en  temps 
opportun.  En  consequence,  une  grande  partie  de  la  recherche  menee  sur  la  PDD  presume  que  les 
processus  de  reconnaissance  sont  cruciaux  a  la  PDD  (voir,  p.  ex.,  Brehmer,  1990;  Gonzalez,  Lerch, 
&  Lebiere,  2003).  Parmi  les  processus  lies  a  la  reconnaissance,  on  retrouve  la  reconnaissance  des 
tendances,  I ’integration  des  indices,  les  heuristiques  et  les  biais. 

Dans  un  effort  visant  a  comprendre  les  defis  particuliers  propres  a  ces  processus,  ce  projet  examine 
les  ecrits  lies  au  domaine  de  la  psychologie  sur  les  mecanismes  cognitifs  sous-jacents  a  la  PDD. 
Plus  precisement,  les  objectifs  de  cette  etude  etaient  d'identifier  et  d'examiner  la  documentation 
pertinente  portant  sur  le  role  de  la  reconnaissance  des  tendances,  Integration  des  indices,  les 
heuristique  et  les  biais  en  relation  avec  la  PDD  et  plus  particulierement  d'identifier  les  defis  et  les 
defaillances  du  raisonnement  normatif;  d'etudier  plus  en  profondeur  les  ecrits  sur  un  domaine  en 
particulier;  de  construire  (ou  d'elaborer  a  partir  d'un  base  existante)  un  cadre  conceptuel  pour  Her  le 
domaine  choisi  a  la  PDD,  dresser  la  Iiste  des  lacunes  de  la  documentation,  etablir  les  domaines  de 
recherches  futures  et  identifier  les  contraintes  et  les  implications  pour  la  formation  en  PDD. 

Ce  projet  etait  divise  en  deux  phases.  La  Phase  I  consistait  en  une  breve  recension  des  ecrits  sur  la 
reconnaissance  des  tendances,  l’integration  des  indices,  les  heuristiques  et  les  biais  en  relation  avec 
la  PDD.  Un  total  de  60  articles  a  d’abord  ete  examine  afin  de  repondre  aux  exigences  de  la  Phase  I. 
A  l’interieur  de  ces  quatre  domaines,  les  themes  generaux,  les  defis  et  les  implications  au  niveau  de 
la  formation  ont  fait  l’objet  de  discussion. 
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Apres  une  breve  analyse,  Pintegration  des  indices  a  ete  choisie  le  domaine  d’interet  particulier  de 
la  Phase  II.  Dans  cette  phase,  17  nouveaux  articles  touchant  precisement  a  Pintegration  des  indices 
ont  ete  etudies.  Les  resultats  de  cette  etude  indiquaient  que  la  documentation  sur  Integration  des 
indices  etait  rare  dans  le  domaine  de  la  PDD.  Une  quantite  importante  de  documentation  relative 
aux  contextes  de  prises  de  decisions  plus  simples  a  done  ainsi  ete  utilisee  pour  induire  le  role  de 
integration  des  indices  dans  la  PDD.  On  estime  que  des  aspects  particulars  de  Integration  des 
indices  —  e'est-a-dire  la  perception,  la  memoire  de  travail,  la  memoire  a  long  terme  et  la 
retroaction  —  jouent  un  role  important  dans  la  PDD.  La  discussion  sur  les  implications  au  niveau  de 
la  formation  comprend  notamment  des  recommandations  sur  ce  qui  doit  faire  l'objet  de  la 
formation  et  la  fa9on  de  former  les  gens.  Le  document  se  termine  par  un  commentaire  de 
conclusion  et  des  recommandations  sur  les  recherches  a  venir  axees  sur  les  FC. 

Dans  Pensemble,  ce  projet  sert  a  expliquer  les  mecanismes  cognitifs  sous-jacents  de  la  PDD  en  tant 
qu’element  contribuant  a  determiner  la  meilleure  fa9on  d’ameliorer  le  processus  de  PDD  et  la 
formation  dans  les  FC  de  Pavenir. 
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1.  Introduction 


1 .1  Background 

The  Canadian  Forces  (CF)  are  increasingly  called  to  serve  in  operations  where  commanders  and 
staff  are  required  to  apply  high-level  dynamic  decision  making  skills.  In  order  to  support  these 
skills,  Defence  Research  and  Development  Canada  (DRDC)  Toronto  is  interested  in  researching 
training  techniques  to  prepare  CF  members  for  deployment  in  complex,  dynamic  environments. 
These  environments  are  characteristic  of  military  operations  such  as  effects-based  operations, 
stability  operations,  counter-IED,  and  counter-insurgency.  It  is  believed  that  the  complex  and 
dynamic  nature  of  these  various  types  of  operations  pose  specific  cognitive  challenges  on  the 
decision-making  process  which  are  not  directly  addressed  by  the  current  training  regimen  of 
military  commanders. 

In  the  psychology  literature,  decision  making  in  complex  and  dynamic  environments  is  generally 
referred  to  as  dynamic  decision  making  (DDM)  in  contrast  with  the  more  “static”  decision  making 
approach  traditionally  researched  in  psychology.  The  research  area  of  DDM  is  typically  presented 
as  being  an  entirely  different  field  than  traditional  decision  making.  This  seems  illogical,  thus  there 
is  a  desire  to  explore  established  psychological  research,  in  particular,  related  to  difficulties  and 
challenges  in  decision  making  that  have  been  documented  in  the  “traditional”  decision  making 
literature,  and  exploit  those  findings  for  their  potential  to  identify  areas  of  research  for  training 
DDM. 

DDM  refers  to  situations  that  require  a  series  of  interrelated  decisions  made  in  real  time  with  the 
aim  of  controlling  or  influencing  a  situation.  In  DDM,  the  state  of  the  problem  changes 
continuously,  both  autonomously  and  as  a  consequence  of  the  decision-maker’s  actions,  and  the 
decision-making  environment  is  opaque  (i.e.,  it  is  not  possible  for  a  decision  maker  to  know  all 
aspects  or  variables  of  the  environment,  and  therefore  some  characteristics  of  the  system  must  be 
inferred)  (Brehmer,  1992). 

At  the  heart  of  DDM  lies  the  need  to  take  into  account  multiple  variables  and  integrate  numerous 
pieces  of  information  gathered  over  time,  in  order  to  make  appropriate  decisions  in  a  timely 
manner.  Accordingly,  much  of  the  research  conducted  on  DDM  assumes  that  recognition  processes 
are  crucial  to  successful  DDM  (see,  e.g.,  Brehmer,  1 990;  Gonzalez,  Lerch,  &  Lebiere,  2003). 
However,  the  DDM  literature  does  not  provide  much  information  on  the  cognitive  mechanisms  that 
would  underlie  DDM.  Specifically,  the  literature  on  processes  such  as  cue  integration  and  pattern 
recognition,  inherent  to  recognition  processes,  is  largely  ignored.  Similarly,  very  little  reference  is 
made  to  the  literature  on  classical  decision  making,  including  the  literature  on  heuristics  and  biases. 
As  there  is  no  compelling  reason  to  assume  that  these  cognitive  mechanisms,  which  are  well 
documented  in  “traditional”  decision  making  research,  do  not  play  a  role  in  DDM,  examining  how 
they  intervene  in  DDM  will  likely  improve  our  understanding  of  the  mechanisms  underlying  DDM. 
This  in  turn  will  help  us  begin  to  understand  how  to  improve  DDM  through  training. 

1 .2  Scope  and  Objectives 

In  order  to  understand  and  determine  how  best  to  improve  the  DDM  process,  the  objective  of  this 
work  is  to  evaluate  what  role  pattern  recognition,  cue  integration,  heuristics,  and  biases  play  in  the 
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DDM  process,  and  what  implications  they  can  have  on  training  DDM.  Building  on  a  previous  report 
on  mental  models  in  DDM  (Brown,  Karthaus,  Rehak,  &  Adams,  2009)  we  first  examine  DDM  as  it 
relates  to  information  processing,  and  DDM  challenges.  From  there,  we  conducted  a  broad  literature 
review  of  pattern  recognition,  cue  integration,  heuristics  and  biases  in  relation  to  DDM.  The  aim  was 
to  identify  the  roles  that  each  of  these  four  factors  play  in  DDM,  and  whether  these  four  areas  pose 
challenges  specific  to  DDM.  We  then  focused  our  efforts  and  conducted  a  more  comprehensive 
literature  review  of  cue  integration  and  the  role  it  plays  in  DDM.  The  goal  was  to  explore  cue 
integration  in  depth  with  the  purpose  of  identifying  how  it  relates  to  challenges  in  DDM.  As  training 
can  be  used  to  influence  mental  processes  involved  in  decision  making  (see,  e.g.,  Hubal  et  al.,  2007) 
and  minimize  the  negative  impact  of  the  difficulties  and  biases  involved  in  recognition  processes, 
our  final  objective  was  to  identify  training  challenges  related  to  the  selected  area. 

This  project  was  divided  into  two  phases.  The  goal  of  the  first  phase  was  to  select  and  review  research 
into  the  psychological  concepts  of  cue  integration,  pattern  recognition,  heuristics  and  biases  in  order 
to  better  understand  the  role  that  these  four  concepts  play  in  DDM.  Specifically,  we  examined  aspects 
of  DDM  that  provide  cognitive  challenges  and  that  are  potentially  relevant  for  training. 

In  the  second,  and  final,  phase  of  the  project  one  of  the  four  concepts  (cue  integration)  was  selected 
for  further  examination.  The  goal  of  this  phase  was  to  conduct  a  comprehensive  review  of  articles 
specific  to  cue  integration  to  gain  a  deeper  understanding  of  the  role  it  plays  in  the  DDM  process  as 
well  as  difficulties  it  presents.  Gaps  within  the  literature,  as  it  pertains  to  DDM,  are  also  identified. 

1.3  Outline  of  report 

This  report  describes  the  methods  used  and  outcome  of  a  two-phase  literature  review  of  recognition 
processes  in  DDM.  The  first  phase  involved  a  broad  review  of  literature  on  cue  integration,  pattern 
recognition,  heuristics  and  biases  to  better  understand  the  role  these  four  concepts  play  in  DDM. 

The  second  phase  involved  a  more  comprehensive  literature  review  of  cue  integration  with  the  goal 
of  understanding  how  it  relates  to  DDM  and  the  difficulties  it  can  present  in  a  DDM  environment. 

This  report  has  7  main  sections  that  describe  this  two  phase  literature  review: 

1.  Introduction 

2.  Methods 

3.  Results  of  Phase  I 

4.  Results  of  Phase  II 

5.  Training  Implications 

6.  Conclusions 

7.  Future  Research  Recommendations 

First,  a  brief  introduction  to  dynamic  decision  making  is  provided  below. 

1.4  Dynamic  Decision  Making 

DDM  generally  refers  to  situations  that  require  a  series  of  interrelated  decisions  made  in  real  time 
with  the  aim  of  controlling  or  influencing  a  situation.  Thus  in  military  DDM  environments,  the 
situation  changes  continuously,  in  a  complex,  hard  to  predict  manner.  It  may  change  autonomously 
and  as  a  result  of  a  decision-makers5  actions.  In  these  types  of  environments  it  is  not  possible  for 
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the  decision  maker  to  know  all  aspects  of  the  variables  (e.g.,  of  the  task,  enemy  and  environment) 
involved.  Inevitably,  some  characteristics  of  the  system  must  be  inferred  (Brehmer,  1992). 

To  further  characterize  DDM  we  identify  four  defining  elements  therein  (Brehmer  &  Allard,  1991; 
Busemeyer,  1999;  Clancy  et  al.,  2003;  Fu  &  Gonzalez  2006): 

•  There  are  a  series  of  decisions; 

•  These  decisions  are  interdependent; 

•  The  environment  changes  both  autonomously  and  as  a  function  of  the  decision  maker’s 
actions;  and, 

•  Timing  is  a  key  element,  where  decision  makers  have  little  control  over  exactly  when 
dynamic  decisions  must  be  made  (Brehmer,  2000). 

In  the  CF,  these  decisions  are  made  within  a  complex  environment  in  attempts  to  achieve  one  or 
more  tasks  (Brown  et  al.,  2009).  Within  these  decisions  a  number  of  cognitive  challenges  are  often 
encountered.  Challenges  concerning  general  cognitive  capacities  and  limitations  (e.g.,  perception), 
the  DDM  tasks  (e.g.,  information  uncertainty),  and  the  DDM  environment  (e.g.,  feedback  delays) 
are  most  relevant.  Specific  challenges  which  appear  to  be  unique  to  DDM  are  described  in  the 
sections  the  follow. 

1.4.1  Challenges  Unique  to  Dynamic  Decision  Making  (DDM) 

DDM  involves  multiple,  interdependent  decisions  within  a  complex,  changing  environment  where 
timing  is  a  key  element  that  effects  the  decision  maker’s  control  over  decisions  and  outcomes 
(Brehmer,  1992,  2000;  Brown  et  al.,  2009). 

There  are  multiple  characteristics  in  DDM  which  can  be  cognitively  challenging  (Brown  et  al., 
2009).  Some  which  are  particularly  challenging  include  uncertainty,  time  pressure,  high  levels  of 
risk,  ill-structured  problems,  complexity,  fast  rate  of  change,  feedback  delays  and  lack  of  adequate 
feedback,  distributed  decision  making  and  making  predictions  (forecasting).  Each  of  these 
challenges  is  discussed  below,  and  relevance  to  military  DDM  is  introduced.  In  later  sections  we 
will  discuss  how  these  particular  challenges  interact  with  the  “traditional”  decision  making 
mechanisms  presented  above. 

1. 4.1.1  Uncertainty 

There  are  many  possible  types  of  uncertainty  which  apply  in  DDM,  some  of  which  are  covered  in 
other  points  (i.e.,  ill-structured  problems  and  lack  of  adequate  feedback).  Some  other  aspects  of 
uncertainty  include  lack  of  knowledge  about  some  important  variables,  lack  of  knowledge  about 
the  validity  of  information,  and  lack  of  knowledge  about  when  new  information  can  be  acquired.  It 
is  generally  challenging  to  make  decisions  without  adequate  information  available,  and  the  decision 
maker  will  have  to  compensate  for  uncertainty  by  either  making  a  decision  with  incomplete 
information  (increasing  risk  of  error)  or  delaying  making  a  decision  (increasing  time  pressure  and 
the  likelihood  that  adequate  intervention  in  a  situation  may  no  longer  be  possible).  Understanding 
the  implications  of  making  decisions  with  incomplete  information  or  delaying  decision  making 
requires  understanding  a  great  many  factors  and  is  highly  challenging. 

Uncertainty  is  a  key  component  of  military  DDM.  Often,  the  primary  function  of  military  actions  is 
to  reduce  uncertainty,  such  as  reconnaissance  operations  and  other  intelligence-gathering  missions. 
Even  when  gathering  information  is  not  the  primary  purpose  of  a  mission  (e.g.,  combat  missions),  a 
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great  deal  of  information  is  often  unknown  (e.g.,  the  exact  location  of  the  enemy  forces;  the  exact 
weapons  that  the  enemy  possesses).  Thus,  training  military  personnel  how  to  deal  with  uncertainty 
during  decision  making  is  critical  for  successful  performance. 

1.4.1. 2  Time  Factors 

Time  is  often  limited  in  DDM,  both  in  terms  of  acquiring  information  and  making  the  decision 
itself.  Time  pressure  creates  a  demand  to  acquire  information  quickly,  integrate  that  information 
quickly,  make  inferences  about  the  information  within  various  time  constraints,  and  determine  an 
appropriate  course  of  action  quickly.  Thus,  time  pressure  puts  a  higher  level  of  demand  on  all 
aspects  of  decision  making. 

There  are  situations  in  military  DDM  where  decisions  have  to  be  made  extremely  quickly  (e.g., 
how  best  to  respond  when  under  attack).  However,  time  factors  in  military  DDM  are  not  limited  to 
decisions  which  have  to  be  made  quickly;  military  scenarios  can  involve  very  long  time  horizons. 

In  circumstances  with  long  time  horizons  (e.g.  military  strategic  planning),  it  can  be  challenging  to 
understand  the  relationships  between  events  and  correctly  determine  causal  linkages.  It  is  also 
important  to  note  in  military  DDM  that  there  are  time  windows  in  which  actions  could  have 
optimal  effects,  and  these  time  windows  can  be  very  short  (e.g.,  a  military  target  is  within  range  for 
only  a  short  period  of  time).  So,  in  military  DDM  contexts,  personnel  have  to  be  able  to  deal  with 
short  and  long  time  horizons,  and  also  be  able  to  detect  the  optimal  time  to  act. 

1.4. 1.3  High  levels  of  risk 

It  has  been  found  that  the  level  of  risk  can  change  how  decisions  are  made;  risks  are  often 
especially  high  in  military  DDM.  Notably  in  military  DDM  contexts  there  are  opportunities  which 
can  easily  be  lost  and  threats  which  will  come  into  play.  The  cost  of  losing  opportunities  or 
experiencing  negative  events  can  change  the  focus  of  decision  makers  and  the  way  that  problems 
are  conceptualized.  These  can  lead  to  biases  in  decision  making.  As  well,  the  level  of  stress 
experienced  in  military  situations  can  place  more  burdens  on  the  decision  maker,  requiring  them  to 
spend  resources  on  dealing  with  the  stress  (theirs  and  others,  particularly  for  leaders)  and  reduce 
the  cognitive  resources  available  for  making  other  decisions.  Military  training  often  involves  high 
stress  and  therefore  CF  personnel  are  generally  familiar  with  operating  in  high-risk  situations; 
however,  training  should  also  take  into  consideration  how  risk  perception  actually  influences 
decision  making.  Training  should  ensure  that  personnel  understand  and  can  mitigate  any  decision¬ 
making  problems  that  are  likely  to  surface  in  high-risk  (or  low-risk)  situations. 

1.4. 1.4  Ill-structured  problems 

Decision  makers  in  DDM  often  are  managing  multiple  goals.  This  means  that  the  current 
conceptualization  of  goals,  prioritization  of  those  goals,  and  the  assessment  of  the  current  situation 
(and  the  impact  on  those  goals)  will  influence  what  decisions  should  be  made  at  a  given  point  in 
time.  Any  shift  in  any  of  these  factors  will  influence  the  current  goal  to  be  accomplished  and 
therefore  could  change  the  appropriateness  of  decisions.  Managing  multiple  goals  is  challenging 
because  there  are  often  many  factors  to  consider,  and  goals  are  often  interdependent,  meaning  that 
moving  toward  accomplishing  one  goal  will  often  impact  other  goals. 

Managing  multiple  goals  is  ubiquitous  in  military  DDM.  A  good  example  is  in  submarine 
operations;  submariners  have  to  constantly  and  simultaneously  manage  the  goals  of  maintaining  the 
safety  of  the  crew,  maintaining  the  safety  of  the  platform,  maintaining  covertness,  and 
accomplishing  mission  goals.  In  military  DDM,  ill-structured  problems  could  also  come  about 
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because  of  direction  from  others  in  a  leadership  position  (i.e.,  commander’s  intent).  If  directions 
are  not  clear  or  understanding  of  those  directions  is  incomplete,  cognitive  resources  have  to  be 
spent  clarifying  what  the  problem  is  so  that  a  solution  can  be  found,  which  directs  resources  away 
from  making  other  decisions.  Military  training  must  ensure  that  CF  personnel  are  comfortable  with 
managing  multiple  goals  and  dealing  with  ill-structured  problems. 

1.4.1. 5  Complexity 

Most  DDM  environments  involve  many  different  factors,  and  this  can  make  decision  making 
complex  (although  how  to  operationally  define  complexity  is  not  clear  in  the  literature).  According 
to  Brown  et  al.  (2009),  the  complexity  of  an  environment  is  considered  relative  to  the  capacity  of  a 
human  to  control  that  environment.  Aspects  of  the  environment  that  could  be  controlled  include  the 
processes,  goals,  action  alternatives,  side  effects,  and  feedback.  Increases  in  the  number  of  things 
that  must  be  controlled,  as  well  as  lags  and  other  factors  which  impair  the  ability  of  decision 
makers  to  understand  the  environment  and  make  good  decisions,  tend  to  increase  complexity. 

One  problem  with  decision  making  in  complex  environments  is  determining  which  factors  are 
important  for  making  a  decision  and  selecting  these  from  the  many  factors  which  are  irrelevant  or 
non-predictive.  The  necessity  of  dealing  with  and  classifying  many  different  pieces  of  information, 
along  with  the  requirement  of  acquiring  this  information,  can  be  cognitively  overwhelming.  The 
literature  shows  that  task  effects  such  as  complexity  and  goal  conflict  are  negatively  associated 
with  performance;  however,  unexpected  interactions  between  such  variables  have  been  found.  For 
example,  Elser  (1993)  found  that  the  highest  levels  of  task  complexity  and  goal  conflict  produced 
the  worst  performance,  but  the  highest  level  of  goal  conflict  produced  better  performance  than  an 
intermediate  level  of  goal  conflict  with  lower  and  intermediate  levels  of  task  complexity. 

Most  environments  in  militaiy  DDM  are  complex,  with  decision  makers  having  to  integrate  many 
different  kinds  of  information  in  a  context  with  an  abundance  of  information  that  may  or  may  not 
be  relevant  for  the  decisions  they  are  trying  to  make.  Military  decision  makers  often  speak  of 
dealing  with  incoming  information  as  “trying  to  drink  from  a  fire  hose”.  Add  to  the  complexity 
produced  by  an  overabundance  of  information  the  fact  that  military  decision  makers  are  often 
responsible  for  making  many  decisions  concurrently  and  it  is  clear  that  training  in  military  DDM 
must  take  complexity  into  consideration. 

1.4.1. 6  Changing  environments 

Many  DDM  contexts  involve  a  situation  that  changes  rapidly,  both  spontaneously  and  because  of 
decision-maker  input.  This  means  that  the  decision  maker  not  only  has  to  gather  initial  values  for 
important  information,  but  also  constantly  update  their  knowledge,  putting  great  demands  on 
attention  and  working  memory.  In  military  DDM  contexts,  change  can  be  extremely  rapid,  and  it  is 
a  major  challenge  keeping  up  with  what  is  occurring  and  the  impact  on  plans.  For  example,  in  a 
combat  situation  both  friendly  and  enemy  forces  manoeuvre,  and  it  is  generally  important  to  track 
the  location  of  your  own  and  enemy  forces,  both  in  an  absolute  sense  (e.g.,  if  you  want  to  call  for 
assistance)  and  a  relative  sense  (e.g.,  if  you  are  attempting  to  fire  on  the  enemy).  Training  must 
include  training  in  how  to  deal  with  highly  dynamic  environments  if  personnel  are  to  effectively 
operate. 

1.4.1. 7  Feedback  delays  and  lack  of  adequate  feedback  for  learning 

Both  feedback  delays  and  lack  of  adequate  feedback  for  learning  are  often  aspects  of  DDM 
environments.  Without  adequate  feedback  it  is  veiy  difficult  to  learn  the  relationships  between 
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inputs  and  outcomes,  as  well  as  information  such  as  cue  validity.  Receiving  appropriate  feedback 
and  interpreting  it  correctly  is  a  major  problem  in  military  DDM.  It  is  often  difficult  to  gain  access 
to  ground  truth  (i.e.,  the  actual  state  of  the  environment),  and  there  are  often  long  delays  between 
actions  and  their  consequences  (e.g.,  in  psychological  operations  effects  may  take  a  long  time  to 
manifest),  or  at  least  knowledge  of  their  consequences  (e.g.,  it  may  take  a  long  time  to  know  what 
the  effects  of  a  bomb  were,  even  though  those  effects  occurred  quickly).  Military  decision  makers 
must  learn  how  to  interpret  the  environment  and  gain  knowledge  about  the  effects  of  their  actions 
and  the  actions  of  others  in  order  to  make  the  best  decisions  as  well  as  to  gain  more  knowledge  in 
the  field. 

1.4.1. 8  Making  predictions  (forecasting) 

One  of  the  key  things  that  decision  makers  must  do  to  be  successful  in  a  DDM  context  is  make 
predictions,  both  about  the  current  direction  of  a  situation  without  input,  and  also  on  the  likely 
impact  of  actions  by  the  decision  maker.  This  is  an  incredibly  challenging  task  in  most  cases, 
largely  because  of  issues  already  discussed  above,  including  the  fact  that  the  situation  involves 
nonlinearities  in  the  variables  to  be  forecasted  that  are  rapidly  changing  and  difficult  to  learn.  All  of 
these  problems  combine  to  make  forecasting  extremely  challenging. 

One  of  the  major  tasks  in  military  DDM  is  to  make  predictions.  Military  operations  such  as 
reconnaissance  and  surveillance  are  intended  to  gather  information  about  the  current  situation; 
however,  the  real  power  of  this  knowledge  is  to  use  this  information  about  current  and  past  events 
to  determine  what  future  events  are  likely  to  occur.  Because  the  majority  of  strategic  and 
operational  military  decision  making  functions  are  done  far  in  advance,  the  accuracy  of  these 
forecasting  functions  is  critical  for  supporting  these  long  time  horizon  activities.  Thus,  it  is 
incredibly  important  that  military  decision  makers  learn  to  make  effective  predictions. 
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2.  Phase  I 


A  structured  process  was  applied  in  conducting  the  Phase  1  literature  review.  The  process  included 
identifying  keywords,  searching  relevant  databases  using  the  keywords,  finding  relevant  articles 
through  extensive  searching,  filtering  relevant  articles,  reviewing  the  articles  and  then  performing 
the  final  analysis  and  synchronization  of  the  literature.  More  details  on  the  process  used  to  search 
for,  identify  and  filter  relevant  articles  can  be  found  below. 

2.1  Phase  I:  Identification,  selection,  and  review  of  articles 

To  structure  the  literature  search,  a  keyword  list  was  created.  This  list  was  sent  to  the  Scientific 
Authority  for  feedback  after  which  minor  changes  were  made.  These  keywords  were  then  used  to 
search  relevant  databases.  They  keywords  used  in  the  search  can  be  found  below  in  Table  1 .  The 
databases  searched  using  the  keywords  can  be  found  in  Table  2. 

To  maintain  a  record  of  the  process,  the  following  information  was  documented  in  a  spreadsheet 
throughout  the  search  process: 

•  Database  searched  (e.g.,  PsyclNFO); 

•  Keyword  combination  (e.g.,  cue  integ*  AND  decision); 

•  Number  of  hits; 

•  Number  of  possible  articles; 

•  Articles  downloaded;  and 

•  Articles/books  that  require  purchase. 

Relevant  literature  was  collected  based  on  abstracts  available  at  the  time  they  were  found.  At  this 
stage  there  were  374  titles  of  relevant  literature.  All  abstracts  were  complied  into  a  database  and 
reviewed  by  the  research  team.  Each  abstract  was  ranked  A,  B,  C,  or  D  based  on  its  relevance  (A 
being  highly  relevant  and  D  being  not  relevant  at  all).  The  research  team  collaborated  to  reach 
consensus  on  the  evaluation  of  each  abstract.  A  total  of  27  pieces  of  literature  were  ranked  as  A, 
having  very  high  relevance,  and  27  as  B,  having  high  relevance.  The  complete  database  was  then 
provided  to  the  SA  who  provided  feedback  on  the  rankings  and  literature  collected.  From  this,  55 
titles  (including  articles  and  books)  were  identified  as  very  highly  relevant  and  should  be 
considered  for  review,  25  were  identified  as  relevant  and  should  tentatively  be  considered  for 
review  if  required,  and  the  remaining  articles  were  noted  as  less  relevant  and  should  not  be 
considered  for  inclusion  in  this  project. 
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Table  1:  Keywords  used  in  Phase  I 


Core  Concept 

Primary  Keywords 

Secondary  Keywords 

Decision  Making 

reasoning,  judgement,  problem 
solving,  intuitive,  rational,  naturalistic, 
choice,  assessment,  evaluation, 
information  processing 

Creativity,  deductive,  inductive,  inference, 
analytic,  counter-factual 

Dynamic  Decision  Making 

risk,  complex*,  uncertain*,  ambiguity, 
problem  space,  opacity,  feedback 
delays,  misperception  of  feedback 

D3M  (distributed  dynamic  decision 
making), 

Systems  theory 

system  dynamics,  complex  systems, 
emergence,  emergent  behaviour 

Nonlinearity,  stock  and  flow,  feedback,  causal 
loop 

Cognitive 

mental,  knowledge,  intellectual,  ability, 
logic,  process*,  model,  sensory, 
perception,  attention,  workload, 
learning,  memory,  language, 
communication 

training,  forgetting,  retention,  recall, 
maintenance,  CECA,  OODA,  recognition- 
primed,  support  theory 

Cue  integrat* 

cue,  store,  identif* ,  together, 
information,  combin*,  indicat*,  event, 
data  fusion 

Pattern  recognit* 

classif*,  instance  based  models, 
memory,  mental  model,  judgment, 
choice,  trend  analysis 

Heuristics 

availability,  representative,  affect,  take- 
the-best,  anchoring  and  adjustment, 
rule  of  thumb,  assumptions,  fast  and 
frugal 

Biases 

error,  systematic,  illogical,  fallacy, 
probability  judgement,  preconception, 
human,  motivation,  misuse 

Statistical,  non  statistical,  confirmation, 
conjunction  fallacy,  base  rate,  regression  to 
the  mean,  congruence,  overconfidence, 
framing,  loss  aversion,  hindsight,  illusory 
correlation,  response  bias,  illusion  of  control, 
clustering  illusion,  optimism,  fundamental 
attribution  error,  egocentric,  false  consensus 
effect,  halo  effect,  in-group  stereotypes, 
prejudice,  herd  instinct,  just-world 
phenomenon,  self-serving 

Contexts 

military,  emergency,  fire  fighting, 
search  and  rescue,  crisis,  operational, 
strategic,  command  and  control,  C2, 
effects  based  operations,  2nd/3rd 
order  effects,  planning,  nuclear,  police 

Uncertain,  critical,  pressure,  risk 

A  variety  of  databases  were  searched.  These  are  summarized  below  in  Table  2. 
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Table  2:  Databases  searched 

Database 

Description 

PsycINFO 

The  PsycINFO  database  is  a  collection  of  electronically  stored  bibliographic  references,  often  with 
abstracts  or  summaries,  to  psychological  literature  from  the  1800s  to  the  present.  The  available  literature 
includes  material  published  in  50  countries,  but  is  all  presented  in  English.  Books  and  chapters  published 
worldwide  are  also  covered  in  the  database,  as  well  as  technical  reports  and  dissertations  from  the  last 
several  decades. 

NTIS 

NTIS  is  an  agency  for  the  U.S.  Department  of  Commerce’s  Technology  Administration.  It  is  the  official 
source  for  government  sponsored  U.S.  and  worldwide  scientific,  technical,  engineering  and  business 
related  information.  The  400.000  article  database  can  be  searched  for  free  at  the  www.ntis.aov.  Articles 
can  be  purchased  from  NTIS  at  costs  depending  on  the  length  of  the  article. 

CISTI 

CISTI  stands  for  the  Canada  Institute  for  Scientific  and  Technical  Information.  It  is  the  library  for  the 

National  Research  Council  of  Canada  and  a  world  source  for  information  in  science,  technology, 
engineering  and  medicine.  The  database  is  searchable  on-line  at  cat.cisti.nrc.ca.  Articles  can  be  ordered 
from  CISTI  for  a  fee  of  approximately  $1 2. 

STINET 

STINET  is  a  publicly-available  database  (stinet.dtic.mil)  which  provides  access  to  citations  of  documents 
such  as:  unclassified  unlimited  documents  that  have  been  entered  into  the  Defence  Technology  Technical 
Reports  Collection  (e.g.,  dissertations  from  the  Naval  Postgraduate  School),  the  Air  University  Library 

Index  to  Military  Periodicals,  Staff  College  Automated  Military  Periodical  Index,  Department  of  Defense 
Index  to  Specifications  and  Standards,  and  Research  and  Development  Descriptive  Summaries.  The  full- 
text  electronic  versions  of  many  of  these  articles  are  also  available  from  this  database. 

Google 

Scholar 

The  World  Wide  Web  was  searched  using  the  Google  Scholar  search  engines  (scholar.google.com). 

DRDC 

Research 

Reports 

DRDC  Defence  Research  Reports  is  a  database  of  scientific  and  technical  research  produced  over  the 
past  6-  years  by  and  for  the  Defence  Research  &  Development  Canada.  It  is  available  online  at 
pubs.drdc-rddc.gc.ca/pubdocs/pcow1_e.html. 

After  this  selection  process  was  complete,  an  initial  60  articles  were  reviewed  by  the  research  team. 
The  articles  were  selected  based  on  the  Scientific  Authority’s  recommendations  and  also  the 
availability  of  the  articles.  Each  article  was  summarized,  and  a  short  critique  of  the  article  was 
performed.  For  each  article,  the  relevant  core  concept(s)  (cue  integration,  pattern  recognition, 
heuristics,  and/or  biases)  were  identified  and  recommendations  whether  to  include  the  article  in  the 
final  literature  review  and  analysis  were  provided.  The  template  used  to  summarize  the  articles  is 
provided  in  Table  3: 
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Table  3:  Template  used  for  article  reviews 


Title:  <> 

Author(s):  <> 

Source:  <> 

Year:  <> 

Abstract:  <> 

Notes: 

<A  paragraph  including  one  to  two  sentences  on  each  of  the  following:  the  main  purpose  of  the  article,  conclusions, 
criticisms/limitations  of  the  article  (if  there  are  any),  brief  evaluation  of  the  article. 

The  content  of  this  “analysis”  paragraph  needs  to  be  geared  specifically  towards  identifying  cognitive  and/or  training 
challenges  in  DDM.  For  instance,  it  could  include  a  brief  analysis  of  how  the  findings/conclusions/models  reported  in  the 
article  identify  or  document  cognitive/training  challenges,  how  these  challenges  are  relevant  for  DDM,  and  if  there  are 
any  limitations  to  the  article  with  regards  to  how  it  can  apply  to  DDM.  Basically  the  idea  of  the  lit  review  in  the  first  phase 
is  to  extract  implications  for  cognitive  and/or  training  challenges  in  DDM,  so  that  we  have  an  idea  of  what  area  is 
particularly  worth  pursuing  further  in  the  second  phase> 

AREA:  CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  <HEURISTICS,  BIASES,  CUE  INTEGRATION,  AND/OR 
PATTERN  RECOGNITION 

To  be  used?  -  Select  one  of  the  following: 

“Include,  if  relevant”  -  These  are  well  written  articles  that  we  think  would  be  excellent  additions  to  the  second  phase  of 
the  project  if  the  area  (e.g.,  biases)  written  about  in  the  article  is  to  be  pursued. 

“For  consideration”  -  These  are  interesting  articles,  that  would  be  satisfactory  additions  to  the  second  phase. 

“Do  not  include”  -  These  articles  are  either  poorly  written,  provide  no  insights  or  are  completely  irrelevant  to  the  project. 


All  article  summaries  are  presented  in  Annex  A.  These  summaries  include  a  general  review  of  each 
topic  and  overall  findings. 

2.2  Results  of  Phase  I 

Having  reviewed  all  Phase  I  articles  (see  Annex  A),  it  was  found  that  cue  integration,  pattern 
recognition,  heuristics  and  biases  each  contribute  differently  to  the  overall  understanding  of  DDM. 
Below  are  definitions  that  we  used  as  guidelines  in  our  review.  Note  that  there  are  many  different 
definitions  in  the  literature  and  so  these  definitions  are  not  to  be  taken  as  consistently  held  by  other 
authors. 


*  Cue  integration  is  the  process  by  which  information  is  selected  from  the  environment, 
weighted,  and  combined  to  serve  as  the  input  to  a  decision  (see  Sections  4.1.1  and  4. 1 .2  for 
more  of  a  discussion  about  the  definitions  of  “cue”  and  “cue  integration”). 

■  Pattern  Recognition  is  the  process  of  applying  knowledge  about  the  relationships  between 
aspects  of  the  environment  (e.g.,  patterns).  Pattern  recognition  is  usually  thought  to  be  a 
function  of  the  similarity  between  the  state  of  the  environment  and  the  patterns  stored  in 
memory,  which  are  usually  learned  through  experience. 
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■  Heuristics  are  shortcuts  in  decision  making,  generally  designed  to  find  a  decision  that  is 
“good  enough”  (i.e.,  satisficing).  Heuristics  are  often  contrasted  with  analytical  methods 
which  systematically  examine  all  possibilities  and  are  intended  to  arrive  at  an  optimal 
decision  (definition  adapted  from  Adams,  Rehak,  Brown,  &  Hall,  2009). 

■  Biases  are  systematic  (i.e.,  non-random)  errors  in  decision  making  (Adams  et  al.,  2009). 

In  general,  there  seemed  to  be  a  greater  selection  of  literature  about  heuristics  and  biases  and  their 
effects  on  (dynamic)  decision  making  compared  to  pattern  recognition  and/or  cue  integration  (see 
Table  1  below);  although  literature  related  specifically  to  improving  decision  making  through  the 
use  of  heuristics  or  training  to  overcome  biases  is  not  prevalent.  On  the  other  hand,  the  literature 
related  to  pattern  recognition  and  cue  integration  in  (dynamic)  decision  making  was  spotty  at  best, 
although  opportunities  for  training  and  improvement  are  possible,  but  not  fully  explored. 

The  quality  of  the  literature  we  reviewed  varied  considerably.  Much  of  the  literature  we  reviewed 
was  of  high  calibre;  however  application  to  DDM  specifically  is  lacking  in  all  four  areas  as  a 
significant  portion  of  the  literature  focuses  on  simpler  decision  making  tasks.  In  Phase  2  of  the 
project,  these  simpler  tasks  were  used  as  building  blocks  to  explore  various  aspects  of  DDM. 

The  literature  review  revealed  that  cue  integration,  pattern  recognition,  heuristics  and  biases  are 
highly  interrelated.  For  example,  biases  are  tightly  linked  with  heuristics.  Much  of  the  literature 
describes  a  dependent  relationship  in  that  heuristics  inevitably  produce  biases  (Janser,  2007; 
Rothrock  &  Kirlik,  2003;  Adelman  &  Bresnick,  1995).  To  a  lesser  extent,  pattern  recognition  has 
been  found  to  overlap  with  heuristics.  Specifically,  research  suggests  that  the  use  of  a  heuristic  can 
be  considered  a  form  of  pattern  recognition  depending  on  the  context  and  resources  (e.g., 
information;  Cohen,  Thompson,  Adelman,  &  Bresnick,  1999).  Pattern  recognition  was  also  seen  to 
be  related  cue  integration,  whereby  the  integration  of  cues  is  done  prior  to  the  recognition  of 
patterns. 

It  should  be  noted  that  the  literature  identified  and  reviewed  was  dependent  on  the  specific  terms 
used  by  authors.  That  is,  our  literature  search  would  have  identified  research  that  used  key  terms 
such  as  “pattern”,  “cue”,  “heuristic”,  and  “bias”  rather  than  other  synonyms.  Acknowledging  that 
the  definitions  for  each  of  these  terms  overlap  and  are  highly  variable,  the  project  teams  did 
specifically  attempt  to  see  through  the  “words”  and  note  relevant  “concepts”. 

Annex  B  contains  summaries  of  the  general  themes  of  each  of  the  four  areas,  specific  cognitive 
challenges  related  to  DDM,  and  training  issues  gleaned  from  the  literature  that  we  reviewed. 

2.3  Selecting  an  area  of  focus 

At  the  end  of  the  first  phase  of  the  project,  cue  integration  was  selected  as  the  concept  to  undergo 
further  examination.  A  number  of  factors  were  considered  in  selecting  this  area  for  future  research 
First,  the  tight  coupling  in  the  literature  between  heuristics  and  biases,  as  well  as  the  cause  and 
effect  relationship  between  them,  meant  that  attempting  to  separate  the  two  concepts  seemed  futile 
(e.g.,  Janser,  2007).  Thus,  heuristics  and  biases  were  merged  and  considered  as  one  category.  This 
reduced  our  consideration  to  three  categories,  from  which  one  needed  to  be  selected  for  more  in- 
depth  research.  Given  that  the  focus  of  this  project  was  on  identifying  trainable  cognitive 
challenges  in  DDM,  we  considered  the  eight  categories  of  challenges  associated  with  DDM  listed 
above  (see  Section  1.4.1).  Then  the  three  remaining  areas  (cue  integration,  pattern  recognition,  and 
heuristics/biases)  were  considered  for  their  relevance  to  the  DDM  challenges  identified.  Table  4 
shows  “yes”  or  “no”  responses  to  the  question  “will  improving  your  skills  in  [category  name]  (e.g. 
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cue  integration)  lead  to  better  decision  making  considering  the  following  challenge  (eg., 
uncertainty)  To  answer  these  questions,  we  used  knowledge  gained  from  Phase  I. 

The  answers  to  the  questions  are  presented  in  Table  4.  In  filling  out  the  table,  it  is  important  to  note 
that  it  was  not  assumed  that  decision  makers  would  have  none  of  these  processes  at  their  disposal 
before  training;  rather,  improving  skills  would  simply  make  the  decision  maker  better  at  them  in 
some  way.  For  example,  it  was  not  assumed  that  training  in  heuristics  would  allow  a  decision 
maker  to  use  heuristics,  but  that  perhaps  using  a  set  of  relevant  heuristics  could  be  made  more 
automatic  so  that  they  would  require  fewer  cognitive  resources.  This  is  because  basic  cue 
integration,  pattern  recognition,  and  heuristics  and  biases  have  been  shown  to  be  used  in  novice 
decision-making,  although  expertise  and  training  may  provide  additional  competence  (e.g.,  see 
Costello,  1992  for  the  use  of  heuristics  in  novices). 

Another  assumption  was  made  that  simply  arguing  that  a  cue  integration  strategy,  pattern  used  in 
pattern  recognition,  or  heuristic  could  be  taught  that  would  be  able  to  address  a  particular  challenge 
(e.g.,  a  heuristic  that  would  deal  with  complexity)  would  not  be  a  useful  or  practical  approach.  This 
is  largely  because  training  would  have  to  be  more  general  (i.e.,  it  would  not  be  known  ahead  of 
time  what  particular  heuristic,  what  cues,  or  what  pattern  should  be  used  by  the  decision  maker). 
The  idea  was  to  assess  which  process(es),  if  generally  improved,  would  show  the  most  promise  in 
addressing  the  DDM  challenge  being  considered. 

Table  4:  Anticipated  improvements  in  cue  integration,  pattern  recognition,  and 

heuristics/biases  in  DDM 


Challenging  DDM 
situations 

Cue 

Integration 

Pattern 

Recognition 

Heuristics/ 

Biases 

Uncertainty 

Yes 

Yes 

No 

Time  factors 

Yes 

No 

No 

High  levels  of  risk 

Yes 

No 

Yes 

Ill-structured  problems 

No 

Yes 

No 

Complexity 

Yes 

Yes 

Yes 

Changing  Environments 

Yes 

No 

No 

Feedback  delays  and  lack 
of  adequate  feedback 

Yes 

No 

No 

Making 

predictions/forecasting 

Yes 

Yes 

Yes 

Total  “Yes” 

7 

4 

3 

Examining  each  of  these  challenges  and  their  effects  on  the  processes  led  to  a  number  of 
conclusions  about  how  well  the  challenges  could  be  addressed  by  the  different  processes. 

The  challenge  of  uncertainty  is  generally  due  to  having  to  make  a  decision  without  sufficient 
information  about  what  the  best  decision  would  be.  Heuristics  are  already  intended  to  result  in  a 
decision  without  using  all  of  the  information  which  might  be  used  (i.e.,  as  compared  with  analytical 
decision  making),  so  there  would  not  likely  be  a  significant  improvement  if  heuristics  were  trained. 
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On  the  other  hand,  both  cue  integration  and  pattern  recognition  tend  to  suffer  without  adequate 
information  (e.g.,  see  Bakken,  2008).  Improving  how  people  seek  out  information  or  what 
information  is  selected  might  improve  performance,  so  training  in  cue  integration  might  be  useful. 
Improving  matching  between  information  in  the  environment  and  increasing  the  number  of  patterns 
in  memory  might  also  improve  performance,  so  training  in  pattern  recognition  might  address  this 
challenge. 

Time  factors  are  a  challenge  in  DDM;  this  can  be  a  problem  because  decision  makers  have  to 
respond  quickly,  or  to  hold  an  action  until  the  ideal  time  window  (or  just  for  a  long  time).  The 
challenge  of  holding  an  action  for  a  long  time  does  not  seem  to  be  well  addressed  by  any  of  the 
three  processes.  Challenges  related  to  making  a  decision  quickly  or  recognizing  that  an  ideal  time 
window  has  been  reached  (which  in  essence  is  also  making  a  decision  quickly)  can  be  addressed  by 
cue  integration.  Cue  integration  is  potentially  a  lengthy  process,  as  it  involves  the  search  for 
information  (e.g.,  see  Famewo  et  al.,  2007).  If  this  process  were  sped  up,  it  is  possible  that  the  time 
factor  challenge  could  be  somewhat  improved.  On  the  other  hand,  both  pattern  recognition  and 
heuristics  may  be  relatively  fast  processes,  and  improving  them  might  improve  the  actual  decision, 
but  it  is  not  clear  whether  that  decision  could  actually  be  sped  up.  For  example,  Wickens  and 
Carswell  (2006)  note  that  decision  making  is  faster  with  familiar  patterns  (i.e.,  context  effects)  and 
heuristics  are  often  used  to  speed  up  decision  making  (e.g.,  Costello,  1992).  Thus,  only  cue 
integration  appears  to  be  a  process  that  could  particularly  benefit  from  training. 

High  levels  of  risk  are  often  a  problem  in  DDM,  both  because  perception  of  risk  can  change 
decision  making  behaviour,  and  also  because  high  risk  tends  to  result  in  stress,  and  dealing  with 
stress  can  limit  the  amount  of  resources  that  a  decision  maker  can  devote  to  decision  making.  If 
decision  making  could  be  made  more  automatic  (i.e.,  use  fewer  resources),  presumably  this  could 
improve  performance.  Pattern  recognition  is  not  assumed  to  require  many  cognitive  resources,  and 
so  it  is  unlikely  that  pattern  recognition  training  would  significantly  lower  the  number  of  resources 
it  uses,  and  so  this  challenge  may  not  be  effectively  addressed  by  pattern  recognition  training.  On 
the  other  hand,  different  heuristics  use  different  amounts  of  resources  (e.g.,  see  Gigerenzer,  2008), 
and  cue  integration  can  be  resource-intensive  (e.g.,  see  Bryant,  2005),  so  training  in  either 
heuristics  or  cue  integration  could  help  to  address  this  challenge. 

Ill-structured  problems  are  a  challenge  because  of  changing  goals;  that  is,  the  task  which  is  to  be 
performed  may  change.  Cue  integration  and  heuristics  typically  require  that  the  task  is  known,  and 
it  is  likely  that  changing  the  task  will  change  what  cues  are  used,  how  they  are  integrated,  and  what 
heuristics  are  appropriate  (e.g.,  see  Juslin,  Olsson  et  al.,  2003).  Thus,  training  in  cue  integration  or 
heuristic  processing  will  not  likely  address  the  challenges  posed  by  ill-structured  problems.  On  the 
other  hand,  it  is  possible  that  pattern-recognition  processes  could  recognize  what  task  is  appropriate 
in  what  situation,  and  therefore  recognize  when  goal  shifting  is  required.  Thus,  training  in  pattern 
recognition  might  address  the  ill-structured  DDM  challenge. 

Complexity,  while  not  well  defined,  is  often  a  challenge  in  DDM.  Different  factors  which  could 
increase  complexity  include  some  challenges  which  are  addressed  separately  (e.g.,  ill-structured 
problems,  changing  environments),  but  there  are  also  other  factors  that  make  a  situation  complex 
(e.g.,  information  overload,  non-linear  relationships).  Interestingly,  complexity  is  a  challenge 
which  may  be  addressed  by  training  in  any  of  the  three  processes.  For  example,  information 
overload  could  be  addressed  by  improving  information-search  processes  through  cue  integration 
training,  by  improving  pattern-matching  processes  through  pattern-recognition  training,  or  by 
reducing  the  amount  of  resources  used  when  applying  heuristics  (so  that  more  resources  are 
available  for  dealing  with  things  like  managing  information)  by  training  heuristic  processing. 
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Changing  environments  is  a  challenge  in  DDM  because  important  information  used  for  decision 
making  can  change  rapidly.  One  way  of  dealing  with  this  challenge  would  be  to  speed  up  decision 
making  so  that  it  can  be  repeated  often  and  therefore  changes  in  the  environment  might  be  detected 
more  quickly.  Thus,  a  similar  rationale  applies  here  as  applies  to  the  time  factors  challenge  above 
(training  cue  integration  appears  to  offer  the  most  potential  to  address  this  challenge  by  increasing 
decision-making  speed). 

Feedback  delays  and  lack  of  adequate  feedback  is  a  challenge  in  DDM  because  feedback  is 
generally  very  important  for  learning.  Treating  feedback  as  a  cue,  training  cue  integration  processes 
to  more  appropriately  deal  with  feedback  could  be  one  way  of  addressing  this  issue.  The  processes 
for  incorporating  feedback  into  pattern  recognition  and  heuristics  would  heavily  overlap  with  cue 
integration  processes  as  we  have  defined  them  for  this  project.  Thus,  it  seems  most  appropriate  to 
argue  that  cue  integration  training  is  the  most  appropriate  for  dealing  with  feedback,  and  that 
training  in  other  aspects  of  pattern  recognition  and  heuristics  would  not  likely  appropriately  address 
the  use  of  feedback. 

Making  predictions  (forecasts)  is  often  a  goal  in  DDM  environments.  That  is,  rather  than 
understanding  a  situation  as  it  is,  decision  makers  often  have  the  (additional)  goal  of  understanding 
how  the  situation  is  likely  to  evolve,  including  how  their  own  actions  might  impact  the 
environment.  Making  inferences  about  the  current  situation  and  how  actions  of  the  decision  maker 
might  change  the  situation  is  a  core  component  of  pattern  recognition.  That  is,  pattern  recognition 
processes  are  intended  to  make  a  match  between  the  current  situation  and  preferred  decisions;  if 
decision  makers  could  be  trained  so  that  pattern  recognition  processes  were  aimed  predicting  future 
rather  than  understanding  current  situations,  this  might  address  this  challenge.  The  focus  of  cue 
integration  is  on  selecting  and  integrating  information  to  be  used  in  a  decision  rather  than  on  the 
output  of  that  decision,  so  it  is  less  likely  that  training  in  cue  integration  would  be  generally  useful 
for  increasing  the  ability  of  a  decision  maker  to  make  predictions.  Likewise,  heuristics  are  shortcuts 
in  decision  making,  and  it  is  not  clear  how  they  could  be  trained  to  increase  the  prediction  accuracy 
of  a  decision  maker.  However,  there  is  evidence  in  the  literature  that  the  task  that  decision  makers 
are  attempting  to  perform  does  change  how  they  integrate  cues  (e.g.,  see  Juslin,  Olsson,  et  al., 
2003),  and  it  also  seems  reasonable  to  suggest  that  which  heuristics  are  used  will  also  change  based 
on  the  task  being  performed  (e.g.,  see  Gigerenzer,  2008).  Thus,  although  training  in  pattern 
recognition  is  the  most  promising  for  addressing  this  challenge,  perhaps  training  in  cue  integration 
or  heuristics  could  also  be  useful  for  making  predictions. 

Thus,  while  all  three  areas  are  suitable  for  further  exploration,  cue  integration  was  determined  to 
have  the  most  potential  for  improving  DDM  given  the  challenges  associated  with  DDM.  (i.e.,  cue 
integration  had  the  most  “yes”  answers  in  Table  4).  Therefore  it  was  agreed  by  the  SA,  members 
of  the  DRDC  research  team  and  members  of  the  HSI  research  team  that  cue  integration  is  the  most 
relevant  to  all  DDM  challenges  discussed  and  that  cue  integration  should  be  the  focus  of  the  second 
phase. 
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3.  Method  of  Phase  II 


In  the  second  phase,  the  first  step  was  to  conduct  a  literature  selection  process,  similar  to  that  in 
Phase  I,  in  order  to  identify  relevant  articles  for  a  more  in  depth  review  of  cue  integration.  A  new 
list  of  key  words  was  generated  and  discussed  with  the  SA  (see  Table  5  below).  These  keywords 
were  then  used  to  search  for  articles  using  the  same  databases  as  for  Phase  I  (see  Table  2). 


Table  5:  Keywords  used  in  Phase  II 


Core  Concept 

Primary  Keywords 

Secondary  Keywords 

Dynamic  Decision  Making 

risk,  complex*,  uncertain*,  ambiguity, 
temporal,  delay*,  feedback,  rate  of 
change,  time  pressure,  ill-structured 
problem*,  distribute*,  predict*, 
forecast* 

judgement,  problem  solving,  intuitive,  rational, 
naturalistic,  NDM,  assessment,  evaluation, 
information  process* 

Cue  integrat* 

data,  store,  identif*,  together, 
information,  combin*,  indicat*,  fusion, 
event,  information  aggregat*,  weight*, 
information  integrat*,  cue  validity 

contingency  learning,  probabilistic  cue*, 

Training 

knowledge,  attention,  workload, 
learning,  memory,  language, 
communication,  assumpt*,  training 
transfer,  skill  retention,  learning 

challenge,  problem 

Contexts 

military,  emergency,  firefighting, 
search  and  rescue,  crisis,  operational, 
strategic,  command  and  control,  C2, 
effects  based  operations,  2nd/3rd 
order  effects,  planning,  nuclear,  police 

uncertain,  critical,  pressure,  risk 

Fifteen  additional  articles  related  to  cue  integration  were  selected  and  again  approved  by  the  SA. 
All  fifteen  articles  were  read  and  summarised.  Summaries  for  each  article  were  completed  using 
the  same  structure  as  outlined  in  Phase  I  (see  Section  2.1).  A  total  of  40  articles  were  examined  in 
more  detail  to  gain  a  deeper  understanding  of  the  role,  difficulties,  and  issues  cue  integration  plays 
in  DDM.  All  findings,  including  gaps  within  the  literature,  as  it  pertains  to  DDM,  are  discussed. 

The  following  sections  provide  our  findings  and  discussions  based  on  the  literature  reviewed  in 
both  Phase  I  and  Phase  II. 
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4.  Results  of  Phase  II 


Through  an  analysis  of  the  literature,  qualitative  assessment,  and  discussion  with  the  SA,  cue 
integration  was  selected  as  the  area  of  focus  for  Phase  II  (see  Methods,  Section  2).  This  section 
provides  an  introduction  to  and  comprehensive  discussion  of  cue  integration,  especially  as  it  relates 
to  DDM. 

4.1  Introduction  to  Cue  Integration 

Before  we  get  into  the  details  of  cue  integration  we  first  address  basic  definitions  as  well  as 
limitations  of  this  work. 

4.1.1  What  is  a  cue? 

The  definition  of  the  term  “cue”  is  an  obvious  starting  point  to  understand  cue  integration.  Shah 
and  Oppenheimer  (2007)  suggest  that  cues  are  the  pieces  of  information  available  to  aid  in  the 
process  of  making  decisions  and  judgments.  Most  other  authors  leave  the  definition  of  a  cue  to  be 
inferred.  For  our  purposes,  a  cue  can  be  considered  any  piece  of  data  used  in  decision  making.  This 
includes  data  about  the  current  situation  (e.g.,  what  is  present  in  the  environment),  data  internal  to 
the  decision  maker  (e.g.,  what  has  been  learned  about  the  task),  and  data  from  the  context  (e.g., 
time  constraints).  Different  types  of  cues  are  potentially  used  differently,  changing  the  resulting 
decision  (Famewo  et  al.,  2007;  Kerstholt  &  Willems,  1993;  Schroeder,  2005).  For  example,  doctors 
will  examine  the  patient,  ask  about  their  family  history  and  look  at  lab  results.  Stockbrokers 
consider  company  profit  margins,  recent  activity  and  quarterly  projections.  In  the  courtroom,  jurors 
include  testimony  from  witnesses  and  evidence  presented  (Shah  &  Oppenheimer,  2007). 

In  further  understanding  cues  we  may  be  inclined  to  look  at  categories  or  “types”  of  cues.  Some 
obvious  categories  may  be  the  information  source  such  as  that  internal  to  the  decision  maker  or 
external  within  the  environment  or  situational  context.  This  distinction,  however,  is  not  clear  within 
the  research;  there  seems  to  be  little  agreement  within  the  literature  of  how  best  to  categorize  cues. 
Cues  may  also  be  categorized  as  task-related  or  context-related.  Our  definition  considers  both 
possibilities  understanding  that  the  source  of  cues  (internal  to  the  decision  maker  vs.  external)  as 
well  as  the  type  of  information  (task  vs.  context-related)  may  impact  the  process  of  cue  integration. 

It  should  be  noted  that  some  researchers  may  identify  cues  as  only  initial  inputs  rather  than  any 
piece  of  data  within  the  decision  making  process.  This  would  in  turn  change  the  way  in  which  cues 
are  discussed  and  significantly  impact  the  interpretation  of  integration.  For  our  purposes  in 
investigating  DDM,  it  seems  most  logical  to  consider  all  data  gathered  within  the  DDM  process  a 
cue.  For  example,  feedback,  by  our  definition,  is  considered  a  cue  to  be  integrated  as  required 
within  a  DDM  context  (e.g.,  Bois,  2002)  as  DDM  environments  are  cyclical  or  continuous  in 
nature.  Others  investigating  simple  decision  making  may  consider  feedback  as  information  that 
contributes  to  learning  rather  than  a  “cue”  as  it  is  not  an  initial  input. 

4.1.2  What  is  cue  integration? 

For  our  purposes,  we  define  cue  integration  as  the  process  by  which  information  is  selected  from 
the  environment,  weighted,  and  aggregated  to  serve  as  the  input  to  the  decision  making  process. 
Further  decomposing  this  definition  helps  to  understand  the  different  aspects  and  components  of 
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cue  integration  necessary  for  our  comprehensive  review.  It  is  quite  challenging  to  identify  a 
recognized  definition  of  cue  integration  as  cues  are  discussed  differently  by  many  different 
researchers.  We  support  Shah  and  Oppenheimer’s  (2007)  definition  of  a  cue  however,  as  it 
identifies  that  all  data  within  DDM  should  be  considered  for  integration  and  utilization  within  the 
decision  as  deemed  appropriate  by  the  decision  maker.  By  our  definition  then,  the  process  of  cue 
integration  spans  the  entire  DDM  process. 

The  literature  review  revealed  that  there  are  many  factors  involved  in  the  way  cues  are  used  in 
decision  making.  First,  there  is  evidence  that  decisions  can  be  influenced  by  the  way  in  which  cues 
are  presented.  For  example,  the  order  as  well  as  the  number  of  cues  presented  at  one  time  can  effect 
emerging  decisions  (e.g.,  Adelman  &  Bresnick,  1995;  Adelman,  Tolcott,  &  Bresnick,  1991).  At 
another  level,  the  relationships  between  cues,  as  well  as  the  relationship  of  the  cue  to  the  criterion 
(desired  goal)  are  very  influential  in  the  use  of  cues  in  decision  making.  Challenges  in  decision 
making  arise  when  there  are  multiple,  complex  relationships.  A  third  factor  is  related  to  cue 
properties.  Information  may  be  unknown  and/or  dynamic.  When  cues  are  affected  by  new  or 
unknown  information,  relationships  pertaining  to  that  cue  (within  the  particular  context)  are 
difficult  to  establish.  Also  relevant  is  the  strength,  or  weight,  of  the  cue.  All  of  these  factors  will  be 
discussed  in  more  detail  in  the  remainder  of  the  report. 

4.1.3  Limitations 

Considerable  limitations  of  this  review  must  be  noted.  Specifically,  there  is  insufficient  literature 
that  integrates  and  discusses  the  mechanisms  of  cue  integration  within  DDM  explicitly.  In 
managing  this  constraint,  the  project  team  utilized  more  simplistic  decision  making  literature  that 
seemed  to  best  relate  to  DDM.  In  order  to  achieve  this  there  was  a  significant  dependence  on  the 
specific  terms  used  by  the  authors  of  the  literature.  Once  the  defining  components  of  DDM  were 
established  they  were  used  in  searching  and  selecting  articles.  In  reviews  investigating  more  simple 
forms  of  decision  making,  aspects  of  DDM  were  sought  out  in  order  to  infer  how  cue  integration 
may  play  a  role  in  a  dynamic  context.  Terms  solicited  pertained  to  “uncertain”  situations;  “time 
pressure”  or  “time  limitations”  tasks;  “high  risk”  situations;  “ill  structured  problems”;  “complex” 
tasks  and  situations;  dynamic  “rate  of  change”;  delays  in  feedback;  and  forecasting  or  “making 
predictions”.  Definitions  of  these  particular  terms  may  vary  thus,  the  project  teams  put  in  specific 
effort  to  see  through  the  “words”  and  note  relevant  “concepts”.  Throughout  this  review  there  are 
interpretations  of  the  role  of  cue  integration  as  it  is  proposed  to  take  place  in  DDM.  Particular 
caution  should  be  taken  when  making  deductions  from  these  findings. 

Overall  the  literature  review  highlighted  the  scarcity  of  literature  that  discusses  the  mechanisms  of 
cue  integration  within  DDM.  From  the  literature  available,  we  therefore  draw  on  Wickens  and 
Carswell’s  (2006)  information  processing  model,  and  DDM  associated  contexts  in  order  to  surmise 
the  role  of  cue  integration  in  DDM.  Such  contexts  include  investigating  high  time  pressure 
environments,  uncertain  situations,  and  complex,  dynamic  tasks. 

4.2  Understanding  Cue  Integration  in  DDM 

The  role  of  cue  integration  in  decision  making,  as  well  as  the  impact  of  its  limitations  on  the 
effectiveness  of  decision  making,  has  been  extensively  studied  in  “traditional”  decision  making 
research.  In  order  to  better  understand  cue  integration,  and  its  role  in  decision  making  it  is  helpful 
to  use  a  model  of  human  information  processing.  The  most  widely  accepted  model  of  information 
processing  is  provided  by  Wickens  and  Carswell  (2006),  and  is  shown  in  Figure  1 :  Model  of 
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human  information  processing 
(adapted  from  Wickens  &  Carswell,  2006). 


Figure  1:  Model  of  human  information  processing 
(adapted  from  Wickens  &  Carswell,  2006) 

This  model  represents  information  processing  as  the  following  set  of  stages: 

1 .  Information  enters  the  information-processing  system  from  the  environment  (this  may  be 
through  several  processes  such  as  unintentional  observation,  searching,  etc.). 

2.  Information  is  processed  by  the  sensory  systems  and  perceptions  are  created  (electrical 
impulses  are  translated  into  information  that  can  be  processed  further,  some  of  which  is 
available  to  consciousness). 

3.  Information  then  enters  into  working  memory,  where  it  is  processed;  this  is  also  called 
short-term  memory  and  holds  a  limited  number  of  items. 

4.  Information  may  move  from  working  memory  into  long-term  memory  if  certain  conditions 
are  present  (e.g.,  information  is  rehearsed).  Likewise,  information  may  move  from  long-term 
to  working  memory.  Long-term  memory  is  a  large-capacity  storage  system  which  can 
maintain  information  for  a  long  duration. 

5.  Information  from  the  processing  in  working  memory  is  evaluated  by  a  response  selection 
system  (e.g.,  there  may  be  a  record  made  of  how  much  evidence  there  is  for  a  particular 
decision,  and  a  criterion  for  when  something  will  be  considered  the  “right”  decision). 

6.  The  response  selection  system  sends  information  into  a  response  execution  system  (e.g., 
the  motor  system)  which  will  translate  the  decision  into  the  appropriate  motor  action. 
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7.  Response  execution  will  have  an  impact  on  the  environment;  this  information  will  then 
feed  back  into  the  sensory  system  and  the  cycle  will  continue. 

8.  Note  that  attention  affects  information  processing  at  many  levels,  including  but  not  limited 
to:  what  information  is  focussed  on  (or  filtered  out  of  processing);  what  information  is 
retrieved  from  long-term  memory  and  what  is  entered  into  long-term  memory  from  working 
memory;  what  responses  are  considered  options,  and  the  effectiveness  with  which  response 
selection  is  translated  into  the  correct  execution. 

Building  on  the  information  processing  model  introduced  above,  the  next  section  discusses  the  role 
of  cue  integration  in  the  various  steps  within  this  model.  Based  on  the  literature  reviewed,  we 
propose  that  cue  integration  is  involved  in  the  information  processing  steps  contained  within  the 
rounded-cornered  box  outlined  below  (blue)  shown  in  Figure  2:  Cue  integration  within  the 
information-Processing  Model 

(adapted  from  Wickens  &  Carswell,  2006).  The  role  of  cue  integration  in  each  of  these  steps  will 
be  discussed  particularly  in  the  context  of  a  DDM  environment.  First,  we  discuss  perception,  then 
attention,  working  memory,  long  term  memory  and  finally,  feedback.  We  propose  that  cue 
integration  is  not  a  primary  component  of  sensory  processing,  response  selection  and  response 
execution  and  therefore  these  steps  are  not  discussed  further.  Sensory  perception  is  what  provides 
inputs  into  perception,  and  therefore  inputs  into  cue  integration.  Conversely  response  selection  and 
eventually  response  execution  can  be  considered  outputs  of  cue  integration. 
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Figure  2:  Cue  integration  within  the  information-Processing  Model 
(adapted  from  Wickens  &  Carswell,  2006) 
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4.3  Attention  and  Cue  Integration  in  DDM 

We  propose  that  attention  is  a  part  of  cue  integration.  Though  there  was  no  literature  found  that 
explicitly  dealt  with  the  role  of  attention  in  cue  integration,  a  number  of  specific  challenges  related 
to  attention  are  discussed.  DDM  contexts  involve  a  situation  that  changes,  both  spontaneously  and 
as  a  result  of  decision-maker  input.  This  means  that  the  decision  maker  constantly  has  to  reassess 
the  current  situation,  putting  great  demands  on  attention  and  working  memory.  Furthermore, 
attention  is  required  for  many  purposes  but  is  a  limited-capacity  system;  this  means  that  parts  of 
decision  making  which  require  attention  (e.g.,  maintaining  information  in  working  memory)  often 
cannot  perform  optimally  (e.g.,  Bois,  2002).  Attention  can  be  directed  in  a  way  that  is  unintended 
by  the  decision  maker  (e.g.,  attentional  capture);  this  means  that  the  information  which  is 
incorporated  into  decision  making  might  not  be  optimal  and  might  not  be  the  information  that  the 
decision  maker  wants  to  influence  the  decision  (e.g.,  Bois,  2002).  Another  challenge  is  that 
attentional  resources  can  fluctuate  with  time  (e.g.,  vigilance  performance  decreases  over  time)  and 
other  factors  such  as  fatigue.  Decision  making  performance,  therefore,  can  be  influenced  by  factors 
such  as  the  amount  of  time  the  decision  maker  has  been  working,  the  context  (e.g.,  how  boring 
their  task  is),  etc. 

One  area  in  which  attention  is  related  to  cue  integration  is  when  the  decision  maker  is  searching  for 
cues.  As  noted,  attentional  resources  can  be  directed  to  particular  cues  unintentionally  (Bois,  2002) 
which  may  result  in  poor  performance.  As  well,  one  difference  between  experts  and  novices  is  that 
experts  seem  to  direct  their  attention  to  (i.e.,  select)  cues  which  are  more  helpful  in  their  decision 
making  (e.g.,  Salas,  Cannon-Bowers,  Fiore,  &  Stout,  2001;  see  Section  5.1  for  further  detail).  At 
this  point,  it  is  unknown  whether  the  selection  of  cues  is  a  deliberate  or  automatic  (i.e.,  conscious 
or  unconscious)  process,  and  this  is  possibly  a  topic  for  future  research. 

4.4  Perception  in  DDM 

A  first  step  of  cue  integration  is  perception  as  cues  must  be  perceived  before  they  can  be 
incorporated  into  the  decision  making  process.  Perception  involves  processing  information 
collected  by  the  sensory  systems.  The  process  of  perception  is  unavoidable;  however,  cue 
perception  is  not  a  straightforward  recording  of  reality.  Instead,  “perception  is  a  choice  of  which 
people  are  not  aware,  and  people  perceive  what  has  been  chosen”  (Kahneman,  2003,  p.  701).  This 
means  that  one’s  perceptions  are  believed  to  be  constructed  (Jones,  2005),  and  that  the  construction 
process  is  affected  by  internal  cues  and  environmental  cues;  thus,  “perception  is  reference- 
dependent”  (Kahneman,  2003,  p.  705).  There  are  large  top-down  processing  influences  of  long¬ 
term  memory  (e.g.,  experience,  expectations)  on  processing  in  the  perceptual  system.  This  is  done 
both  consciously  and  subconsciously1. 

Perception  is  not  perfect  and  is  associated  with  its  own  set  of  challenges.  For  example,  only 
information  which  has  been  processed  by  the  sensory  system  can  be  entered  into  cognitive 
processing.  This  means  that  some  information  can  be  missed,  and  unavailable  information  cannot 
be  used  (e.g.,  Bakken,  2008;  Bois,  2002).  This  is  especially  the  case  in  DDM  situations  where  the 
quantity  of  task  cues  is  large,  and  the  interrelationship  between  cues  can  be  complicated. 


1  The  role  of  consciousness  in  cue  integration  was  not  specifically  explored  in  this  project.  This  could  be  a 
relevant  area  for  future  research. 
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4.4.1  Different  Cue  Types 

Research  suggests  that  there  are  different  types  of  cues.  Some  cues  are  task-related  and  taken  from 
the  environment  and  other  cues  are  internal  to  the  decision  maker.  Each  of  these  cues  is  perceived 
differently.  Schwartz  and  Norman  (1989)  suggest  that  environmental  cues  are  taken  into  account  if 
“a)  it  truly  varies  within  the  ecology,  and  decision  makers  attend  to  this  or  b)  some  aspect  of  the 
situation  surrounding  the  decision  causes  the  perception  of  cue  variation”  (p.  383).  In  both 
circumstances,  the  perception  of  variability  is  what  determines  cue  usage. 

Other  cues  are  internal  to  the  decision  maker  and  include  natural  reactions  to  stress  (perhaps  from 
time  pressure).  Another  major  source  of  internal  cues  is  affect,  or  emotion/mood.  For  example, 
when  making  decisions,  people  sometimes  use  their  current  emotional  state  as  a  cue  in  the 
judgment  process.  Take  the  following  two  questions:  “How  happy  are  you?”  and  “How  many  dates 
did  you  have  last  month?”  The  order  in  which  these  questions  are  asked  has  implications  for  how 
the  questions  are  answered.  Kahneman  (2003)  showed  that  there  was  little  correlation  between 
responses  to  these  two  questions  when  asked  in  this  order;  however,  when  the  order  was  reversed 
the  correlation  rose  to  0.66.  In  other  studies,  the  intensity  of  fear  has  been  found  to  impact  a  variety 
of  judgments,  such  as  rating  the  probability  of  a  disaster  (Kahneman,  2003).  These  simple 
examples  show  a  more  complex  phenomenon;  namely  that  people’s  affect  influences  how  they 
perceive  the  world  and  how  they  make  decisions.  These  studies  point  to  the  concept  that  emotions 
(or  affect)  can  be  considered  cues.  People  performing  in  time  pressured,  complex,  applied  DDM 
environments  should  be  subject  to  these  same  affect  biases  which  can  impact  their  performance. 

See  Section  5.2  for  a  discussion  of  cue  types  in  the  context  of  training. 

4.4.2  Tasking  Effects 

Other  research  has  looked  at  the  effect  of  what  is  being  asked  of  decision  makers  on  cue  integration. 
These  studies  show  that  the  strategies  used  and  conclusions  arrived  at  can  vary  greatly  depending  on 
the  specific  tasking  of  the  decision  maker,  even  though  the  same  cues  are  presented.  Evans,  Clibbens, 
Cattani,  Harris,  and  Dennis  (2003)  found  differences  when  decision  makers  were  asked  to  integrate 
cues  together  and  provide  general  assessments  between  poor  and  good  (called  multicue  judgements) 
versus  when  asked  to  select  between  two  cases  (forced  choice).  It  is  assumed  that  DDM  scenarios 
tend  to  involve  both  of  these  types  of  decision  making.  The  authors  discovered  that  performance  was 
much  better  on  the  standard  multi-cue  judgment  task  versus  the  forced-choice  task. 

In  the  same  series  of  experiments,  Evans  et  al.  (2003)  introduced  cues  that  predicted  criterion 
negatively  (negative  predictors)  and  increased  the  complexity  of  the  task  by  increasing  the  number 
of  cues.  Results  suggest  that  irrelevant  cues  do  not  prevent  people  from  acquiring  linear  judgment 
models  but  simply  render  these  models  less  accurately  matched  to  the  criterion.  The  role  of  implicit 
and  explicit  cognitive  processes  in  these  judgments  was  also  reviewed.  Overall,  the  authors  found 
that  both  implicit  and  explicit  processes  influence  multicue  judgements.  Specifically  insight 
measures  (implicit  processes)  dropped  when  a  multicue  judgment  task  is  more  difficult,  however, 
explicit  knowledge  does  not  appear  to  be  impacted  by  task  difficulty.  Implicit  learning  is  also 
impaired  by  the  presence  of  irrelevant  cues  in  the  environment.  Both  these  effects  suggest  that 
explicit  knowledge  should  continue  to  increase  in  DDM  situations,  though  implicit  learning  may  not. 

In  another  example,  judgments  regarding  the  custody  of  a  child  were  assessed  based  on  parental 
descriptions  (Kahneman,  2003).  One  description  was  much  richer,  in  both  negative  and  positive 
attributes.  Subjects  were  either  asked  to  judge  which  custody  request  should  be  accepted  or 
rejected.  Under  both  “accepted”  and  “rejected”  conditions  the  richer  description  was  selected.  This 
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means  that  the  same  parent  who  was  accepted  in  the  “accept”  condition  was  the  parent  that  was 
rejected  in  the  “reject”  condition.  Clearly  the  subject’s  focus  was  shifted  based  on  the  question 
asked  even  though  the  same  information  was  presented  (Kahneman,  2003). 

Finally,  Juslin,  Olsson,  and  Olsson  (2003)  found  that  different  cognitive  processes  are  used  to 
determine  categorization  (i.e.  allocating  between  two  or  more  categories)  versus  multiple  cue 
judgments  of  a  continuous  variable  (e.g.,  probability  of  precipitation  given  weather  information). 
Based  on  three  experiments,  Juslin,  Olsson  et  al.  concluded  that  differences  between  the 
categorization  and  multiple-cue  judgment  tasks  actually  produce  qualitatively  different  cognitive 
processes.  Mental  cue  abstraction  is  favoured  when  a  continuous  criterion  is  used,  whereas  a  binary 
criterion  tends  to  be  interpreted  via  exemplar  memory. 

In  DDM,  it  is  likely  that  different  types  of  outputs  will  be  appropriate  for  different  decisions.  For 
example,  if  a  risk  assessment  is  being  performed,  it  may  be  appropriate  to  make  a  categorical 
assessment  (e.g.,  an  approaching  individual  is  a  risk  or  not),  or  it  may  be  appropriate  to  make  a 
proportional  judgment  (e.g.,  deciding  the  relative  riskiness  of  entering  different  parts  of  a  city). 
Because  the  tasking  can  influence  decision-making  processes,  DDM  contexts  should  be  examined 
to  determine  potential  tasking  types  and  the  implications  of  these  different  tasking  types  on  the  way 
that  information  should  be  presented  and  the  types  of  processing  that  need  to  be  supported.  As 
well,  consideration  about  how  performance  should  be  evaluated  is  also  needed,  including  whether 
assessment  of  performance  across  different  kinds  of  tasks  is  appropriate  (see  Section  5.3.4  for  a 
discussion  of  evaluating  performance  during  DDM  training). 

4.4.3  Information  Presentation  Effects 

In  addition  to  what  is  asked  of  a  decision  maker,  the  way  in  which  cues  are  presented  to  a  decision 
maker  has  also  been  shown  to  impact  how  they  are  perceived.  The  effects  of  information 
presentation  (e.g.,  framing)  of  cues  on  decisions  have  been  extensively  researched.  The  order, 
format,  salience,  context,  and  affect  of  cues  impacts  how  and  whether  a  cue  is  incorporated  into 
decision  making  (Adelman  and  Bresnick,  1995;  Bois,  2002;  Dunwoody,  Haarbauer,  Mahan, 
Marino,  &  Tang,  2000;  Famewo  et  al.,  2007;  Jones,  2005;  Payne,  Bettman  and  Johnson,  1992; 
Tolcott,  Marvin  and  Bresnick,  1996).  Some  of  these  studies  conclude  that  the  effects  of  framing 
occur  outside  of  consciousness,  such  as  the  tendency  to  judge  absolute  numbers  (e.g.,  20  out  of 
100)  as  more  likely  to  occur  than  probabilities  (20%)  (e.g.,  Slovic,  Peters,  Finucane,  &  MacGregor, 
2005);  while  decision  makers  can  be  made  aware  of  other  effects  such  as  the  tendency  to  be  risk 
seeking  or  risk  adverse  (Schulz,  1996).  Collectively,  these  effects  may  lead  to:  neglecting 
potentially  relevant  problem  information,  ignoring  values  when  they  are  difficult  to  estimate, 
inaccurately  weighting  cue  validity,  and  the  “general  tendency  to  accept  problems  as  given  and  the 
segregation  of  the  decision  problem  at  hand  from  its  broader  context”  (Payne,  Bettman,  &  Johnson, 
1992,  p.  1 16).  In  general,  each  of  these  effects  can  be  expected  to  impact  military  scenarios  as  well 
drawing  attention  to  the  importance  of  the  presentation  of  information  to  military  decision  makers. 
Accessibility  and  gain/loss  framing  are  specifically  discussed  below  in  addition  to  other  general 
information  presentation  effects  (for  a  discussion  of  information  presentation  during  training,  see 
Section  5.3.1). 

4.4.3. 1  Accessibility 

Accessibility  refers  simply  to  “the  ease  (or  effort)  with  which  particular  mental  contents  come  to 
mind”  (Kahneman,  2003,  p.  699).  Accessibility  is  not  a  dichotomy  but  a  continuum  where  some 
operations  demand  more  effort  than  others.  The  more  accessible  a  cue  is,  the  more  it  will  tend  to  be 


Humansystems 


Cue  integration  in  Dynamic  Decision  making 


Page  23 


g 


HUMANS'  'STEMi 


used  in  decision  making  or  other  tasks.  Highly  accessible  cues  can  also  become  steady  anchors.  A 
number  of  perceptual  examples  are  used  to  illustrate  how  certain  characteristics  are  more 
“accessible”  than  others  to  provide  some  perceptual  examples  of  the  effects  of  accessibility.  Figure 
3,  for  example,  shows  how  the  height  and  surface  area  of  blocks  are  more  obvious  (and  hence, 
accessible)  depending  on  the  way  they  are  presented.  It  is  easier  to  see  that  the  volume  is 
equivalent  between  A  and  C  in  Figure  3  because  the  columns  are  the  same  height  and  width, 
compared  to  assessing  the  volume  between  B  and  either  A  or  C.  Thus,  desired  information  can  be 
easier  or  more  difficult  to  perceive. 


Figure  3:  The  selective  accessibility  of  natural  assessments  (Kahneman,  2003) 

In  contrast  to  one  piece  of  information  being  more  or  less  difficult  to  perceive  as  in  the  previous 
example,  it  is  also  possible  that  different  perceptions  of  an  ambiguous  piece  of  information  can  be 
differently  accessible  depending  on  the  context.  Figure  4  shows  an  example  of  how  the  context  in 
which  objects  are  embedded  can  also  influence  the  accessibility  of  different  interpretations. 


Figure  4:  An  effect  of  context  on  the  accessibility  of  interpretations 

(Kahneman,  2003) 
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One’s  interpretation  of  the  middle  figure  in  both  lines  is  dependent  on  the  objects  on  either  side. 

With  the  middle  figure  embedded  between  “A”  and  “C”,  we  are  more  likely  to  see  it  as  a  “B”, 
rather  than  as  a  “13”  (as  in  the  lower  row). 

Salience  also  directly  relates  to  accessibility.  Thus,  if  a  large  green  letter  and  a  smaller  blue  letter 
are  presented  simultaneously,  “green”  will  first  come  to  mind.  This  effect  can  be  mitigated  -  for 
example,  if  told  to  look  for  smaller  letter,  then  the  blue  letters  features  will  be  enhanced.  Kahneman 
and  Frederick  (2002;  as  cited  in  Kahneman,  2003)  listed  natural  assessments  which  they  argue  are 
perceptually  registered  with  little  effort.  These  include  size,  distance,  loudness,  similarity,  causal 
propensity,  “surprisingness”,  affective  valence  and  mood  (Kahneman,  2003). 

The  accessibility  of  cues  can  impact  how  they  are  used  in  decision  making.  Shah  and  Oppenheimer 
(2007)  suggest  that  decision  makers  use  cue  fluency  (i.e.,  how  easily  the  meaning  of  the  cue  is 
accessed)  to  determine  cue  weights  by  “placing  more  weight  on  information  that  feels  easy  to 
process”  (Shah  &  Oppenheimer,  p.  371).  Different  types  of  fluency  are  described,  including 
perceptual  fluency  (i.e.  how  easily  information  is  perceived)  and  retrieval  fluency  (i.e.  how  easily 
information  is  recalled  from  memory).  The  authors  argue  that  the  ease  of  processing  itself  (i.e.,  cue 
accessibility)  becomes  a  cue  that  is  weighted  and  directly  affects  decision  making. 

Cue  fluency  can  also  affect  decision  making  indirectly,  by  influencing  which  decision  making 
strategy  is  selected.  Part  of  the  argument  for  this  is  that  fluency  is  correlated  with  objective  validity 
in  the  real  world,  making  fluency  an  obvious  basis  for  cue  weighting.  Two  experiments  examining 
the  effect  of  perceptual  fluency  were  conducted.  The  first  experiment  had  subjects  view  a  negative 
consumer  review  and  then  suggest  a  price  to  be  paid  for  the  reviewed  product.  Some  subjects  read 
reviews  written  in  smaller  italicized  font  and  the  other  review  was  easier  to  read.  Subjects  were 
willing  to  pay  more  when  the  negative  review  was  more  difficult  to  read,  indicating  that  subjects 
weighed  this  information  less  than  the  fluent,  easy  to  read  review.  The  second  experiment  was  a 
between-subjects  design  that  also  included  easy  and  harder  to  read  information,  though  this  time  the 
information  was  a  rating  score  (either  numeric  or  semantic).  Again,  the  fluent  cue  was  used  more  in 
decision  making  than  the  less-fluent  cue  (Shah  &  Oppenheimer,  2007).  Thus,  experimental  findings 
indicate  that  cues  which  are  more  accessible  appear  to  be  weighted  more  heavily  in  cue  integration. 

The  reasons  for  these  results  are  discussed.  Shah  and  Oppenheimer  (2007)  postulate  that  perhaps  it  is 
an  increase  in  attention  to  the  fluent  items  that  leads  them  to  be  more  heavily  weighted.  Alternatively, 
they  propose  that  “people  simply  select  accessible  or  fluent  information  until  they  feel  that  they  have 
enough  information  and  then  disregard  the  rest  of  the  information”  (Shah  and  Oppenheimer,  p  377). 
Though,  the  clustering  of  results  around  the  midpoint  of  both  the  fluent  and  disfluent  ratings  in  the 
second  experiment  suggest  that  in  fact  both  cues  were  considered;  just  the  fluent  cue  was  more 
heavily  weighted.  Though  not  yet  tested,  Shah  and  Oppenheimer  hypothesized  that  if  the  use  of 
fluency  for  cue  weighting  minimizes  cognitive  effort  then  this  strategy  should  be  applied  more  often 
when  decision  makers  are  placed  in  situations  where  there  are  high  cognitive  demands. 

The  importance  of  accessibility,  salience  and  fluency  on  a  decision  makers’  ability  to  select  cues 
has  wide  ranging  effects  in  DDM.  Notably,  in  complex,  uncertain  military  DDM  environments  it 
can  be  assumed  that  important  cues  are  not  necessarily  going  to  be  salient.  Cue  values  will  also  be 
changing  often  making  the  current  value  of  cues  less  accessible. 

4.4.3.2  Gain  versus  Loss  Framing 

In  addition  to  the  potential  impact  of  accessibility,  intuitive  processes  can  also  be  susceptible  to 
framing  effects.  Depending  on  the  way  in  which  a  problem  is  framed,  respondents  can  be  more  risk 
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averse  or  risk-seeking  even  though  they  are  asked  essentially  the  same  question.  This  phenomenon 
has  been  studied  in  detail  in  the  context  of  a  disease  problem.  This  problem  can  be  summarized  as 
follows  (Tversky  &  Kahneman,  1981): 

“Imagine  that  the  United  States  is  preparing  for  the  outbreak  of  an  unusual  disease,  which 
is  expected  to  kill  600  people.  Two  alternative  programs  to  combat  the  disease  have  been 
proposed.  If  Program  A  is  adopted,  200  people  will  be  saved.  If  Program  B  is  adopted, 
there  is  a  one-third  probability  that  600  people  will  be  saved  and  a  two-thirds  probability 
that  no  people  will  be  saved.  Which  one  of  the  two  programs  would  you  favor?” 

When  presented  with  this  problem,  most  respondents  favor  A  (risk  aversion).  Alternatively,  if 
presented:  “If  Program  A  is  adopted,  400  people  will  die.  If  Program  B  is  adopted,  there  is  a  one- 
third  probability  that  nobody  will  die  and  a  two-thirds  probability  that  600  people  will  die”,  the 
majority  favors  B  (risk  seeking).  This  tendency  is  reinforced  by  Kahneman  (2003):  “A  basic 
principle  of  framing  is  the  passive  acceptance  of  the  formulation  given”  (p.  703). 

The  impact  of  framing  on  decision  making  has  also  been  shown  in  real-world  healthcare  examples 
in  which  outcomes  depend  on  whether  survival  or  mortality  rates  are  shared.  Thus,  healthcare 
decisions  changed  based  on  whether  or  not  patients  were  presented  with  the  survival  rate  or  the 
mortality  rate  (Kahneman,  2003).  It  is  clear  that  the  way  in  which  the  cues  are  being  integrated 
varies  greatly  based  on  the  gain/loss  presentation,  even  though  essentially  the  same  information  is 
being  presented. 

Schultz  (1996)  discusses  how  each  of  these  example  outcomes  can  be  thought  of  as  changes  from  a 
reference  point.  The  reference  point  (which  can  be  thought  of  as  a  cue)  is  itself  changed  by  how  the 
decision  maker  evaluates  and  edits  the  decision  problem.  Editing  and  framing  the  problem  are 
dynamic  processes,  which  can  change  the  assessment  of  a  particular  outcome  from  a  gain  to  a  loss, 
and  vice  versa.  Thinking  of  an  outcome  as  a  gain  or  a  loss  will  influence  the  assessment  of  risk, 
which  will  in  turn  influence  decisions. 

“Gain”  or  “Loss”  therefore  appears  to  be  a  significant  cue  in  decision  making  that  is  easily 
manipulated.  This  finding  supports  the  idea  that  cues  are  commonly  interpreted  in  a  binary  format 
(i.e.,  Yes  or  No,  Gain  or  Loss),  even  for  cues  are  that  continuous  (>50  years  old  or  <50  years  old) 
(Newell  et  al.,  2004).  Applying  the  gain/loss  framing  effect  to  military  applications  would  have  strong 
implications  on  all  levels  of  military  decision  making  from  tactical  up  to  strategic.  For  example,  a 
losing  commander  may  be  inclined  to  accept  more  risks  that  one  who  is  winning  (Schultz,  1996). 

4.4.3.3  Other  Framing  Effects 

There  are  many  other  framing  effects  relevant  to  cue  integration  and  decision  making  that  have 
been  studied.  For  example,  cue  presentation  can  even  determine  the  decision  making  strategy  used 
by  a  decision  maker.  For  example,  Dunwoody  et  al.  (2000)  show  that  different  decision  making 
strategies  were  induced  by  an  iconic  information  display  compared  to  a  numeric  information 
display.  Also,  there  appear  to  be  differences  in  how  cues  are  integrated  depending  on  whether  cues 
are  presented  all  at  once  or  serially.  With  evidence  suggesting  that  better  decisions  are  made  when 
cues  are  presented  simultaneously;  however,  this  likely  depends  on  factors  such  as  number  of  cues 
as  well  as  the  nature  of  cues.  Further,  when  cues  are  presented  serially,  cues  can  be  weighted 
differently,  resulting  in  different  decisions  (e.g.,  Adelman  &  Bresnick,  1995;  Adelman,  Tolcott,  & 
Bresnick,  1991).  Unfortunately,  while  a  great  deal  of  information  can  be  presented  simultaneously 
to  military  decision  makers,  the  mission  scenario  will  continue  to  change.  This  means  that  cues  will 
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not  remain  static  as  their  values,  proper  weight,  etc.  vary.  As  such,  changing  cues  will  be  required 
to  be  also  presented  to  military  decision  makers.  It  is  reasonable  to  presume  that  this  forced  serial 
presentation  of  cues  (and  changing  cues)  contributes  to  the  challenge  of  DDM. 

Additionally,  one  study  explored  the  usage  of  unimportant  cues.  Schwatz  and  Norman  (1989) 
found  that  cues  that  are  unimportant  can  be  considered  relevant  and  used  by  decision  makers.  For 
these  studies,  job  applicant  evaluations  were  performed  by  subjects.  In  their  decisions,  unimportant 
cues  (e.g.,  height)  were  shown  to  be  used  in  decision  making.  The  use  of  these  cues  was  increased 
when  it  was  subtly  suggested  to  subjects  that  the  provided  cues  were  known  to  be  relevant  to  job 
evaluations.  The  authors  argue  that  situation  context  determines  whether  or  not  the  cues  are  used  to 
make  decisions.  In  DDM  environments,  decision  makers  rely  on  various  means  to  suggest 
relevance  and  this  study  shows  how  easily  relevance  can  be  misguided. 

The  second  experiment  included  unavailable  (i.e.,  missing)  and  unknown  (i.e.,  stated  “unknown”) 
cues  as  part  of  a  similar  job  applicant  tasking.  The  authors  hypothesized  that  missing  information 
could  be  utilized  in  three  possible  ways:  1)  participants  infer  the  missing  values;  2)  missing 
dimensions  could  draw  more  attention  than  those  with  values;  and  3)  dimensions  without  missing 
information  could  be  more  heavily  weighted.  Again,  some  participants  received  distinct  instructions. 
They  were  informed  that  “unavailable”  information  was  not  included  and  the  supervisor  did  not  care 
to  know  the  values,  and  further  that  “unknown”  information  was  also  not  included  but  those  values 
were  desired.  Results  showed  that  subject  preference  for  job  applicants  decreased  similarly  with 
unavailable  and  unknown  information;  though  there  was  more  variability  in  preference  ratings  for 
unknown  information  than  for  unavailable  information.  This  is  a  somewhat  surprising  conclusion,  as 
one  would  expect  unavailable  information  to  be  disregarded  and  unknown  information  to  be  heavily 
weighted.  See  Section  5.2  for  a  discussion  of  how  framing  effects  might  be  considered  and  mitigated 
during  DDM  training  (e.g.,  through  critical  thinking  training). 

4.4.4  Perception  Summary 

In  conclusion,  the  preceding  sections  suggest  that  cue  perception  is  not  consistent  or 
commonsensical.  How  cues  are  presented  can  bias  cue  integration,  unimportant  cues  can  be 
considered  in  the  decision  making  process,  and  a  lack  of  cues  (e.g.,  cues  marked  as  “unknown”) 
can  affect  cue  integration  and  decision  making  in  ways  that  one  may  not  expect.  These  results  have 
been  found  in  experiments  using  relatively  simple  judgment  tasks.  One  can  therefore  presume  that 
the  effects  will  be  compounded  in  complex,  uncertain  dynamic  military  environments  that  are 
characteristic  of  DDM. 

A  specific  challenge  in  military  scenarios  involves  perceiving  the  presence  of  critical  cues 
(detection).  In  most  laboratory  experiments,  cues  are  explicitly  presented  to  participants  and 
participants  do  not  have  to  perform  a  visual  (or  other  perceptual  sense)  search  for  cues.  That  is, 
perceptual  detection  tasks  are  often  not  a  part  of  decision  making  tasks.  In  contrast,  in  military 
environments,  decision  makers  are  often  acquiring  information  over  long  periods  of  time,  and 
important  information  must  be  detected  and  can  be  missed  (e.g.,  changes  on  a  visual  display  could 
be  overlooked).  There  is  some  evidence  that  decision  makers  do  miss  perceptual  cues,  and  that 
helping  decision  makers  detect  important  cues  can  improve  decision  making  (e.g.,  Kirlik,  Walker, 
Fisk,  &  Nagel,  1996;  see  Section  5.3.1  for  further  discussion  about  training  perceptual  processing). 

Cues  themselves  are  likely  to  be  much  more  ambiguous  in  military  DDM  scenarios  than  in  the 
laboratory  experiments  which  are  typical  of  the  research  reviewed  here.  Cues  in  laboratory 
experiments  most  often  have  unambiguous  values  which  are  easily  discriminated  (e.g.,  in  Juslin, 
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Jones,  Olsson,  and  Winman’s  (2003)  experiment  in  which  the  toxicity  of  frogs  had  to  be  evaluated, 
frogs  were  either  clearly  brown  or  clearly  green)  but  military  decision  makers  there  are  many 
different  values  which  overlap  and  are  ambiguous  and  thus  much  more  difficult  to  discriminate 
(e.g.,  there  are  green  frogs  with  brown  spots,  brown  frogs  with  green  spots,  greeny-brown  frogs, 
etc.).  Thus,  even  when  a  relevant  item  in  the  environment  is  identified,  it  is  often  a  perceptual 
challenge  to  determine  relevant  information  about  that  item  (i.e.,  is  the  frog  green  or  brown?).  It 
has  been  shown  that  training  in  how  to  set  thresholds  for  continuous  variables  can  improve  DDM 
performance  (see  Section  5.2  for  a  discussion  of  training  implications). 

4.5  Working  Memory  and  Cue  Integration  in  DDM 

Next,  we  examine  the  role  of  cue  integration  in  working  memory  specifically  in  DDM  contexts. 
Working  memory  is  the  powerhouse  of  processing,  where  the  links  between  pieces  of  information 
are  made.  We  assume  that  it  is  in  this  function  that  that  the  main  decision  making  strategies  are 
selected  and  applied.  This  stage  of  processing  has  extremely  limited  capacity.  Working  memory 
performs  both  storage  and  processing  functions  and  these  are  generally  not  independent.  This 
means  that  the  more  things  that  have  to  be  remembered  the  fewer  resources  that  are  available  for 
processing,  and  vice  versa,  which  means  that  information  can  be  lost  and  inferences  can  be  missed. 
Finally,  attention  is  required  to  maintain  information  in  working  memory.  This  means  that  if 
attentional  resources  are  allocated  elsewhere  information  is  likely  to  be  lost.  Below  we  focus  on  the 
number  of  cues  available  and  the  possible  ways  in  which  these  cues  are  weighted  and  integrated 
within  working  memory.  The  role  of  cues  in  mental  models  is  also  discussed. 

4.5.1  Large  quantity  of  cues 

It  has  been  found  in  relatively  simple  decision  making  tasks  that  as  the  number  of  cues  increases, 
performance  drops.  For  example,  one  study  found  that  judgment  accuracy  was  highest  with  two 
cues  and  as  the  number  of  cues  increased  accuracy  decreased  (Karelaia  &  Hogarth,  2008).  In  real 
world  military  dynamic  decision  making  tasks,  however,  there  will  most  certainly  be  more  than 
two  cues;  so  many  more  that  the  sheer  quantity  of  cues  can  exceed  the  cognitive  limitations  of  the 
decision  maker  (e.g.,  Adelman  &  Bresnick,  1995;  Jones,  2005).  The  limitations  of  working 
memory  restrict  the  ability  to  process,  and  integrate  multiple  sources  of  information  (e.g.,  Famewo 
et  al.,  2007;  Gibb,  2000;  Jacobs  &  Gaver,  1998).  This  means  that  links  between  ideas  can  only  be 
made  if  they  are  active  within  working  memory  at  the  same  time.  The  result  could  be  that  many 
inferences  are  missed  (e.g.,  Adelman  &  Bresnick,  1995;  Bakken,  2008). 

Although  it  might  be  expected  that  expert  decision  makers  might  be  better  decision  makers  because 
of  an  ability  to  use  more  cues,  it  has  been  found  that  experts  do  not  use  more  cues  than  novices. 
Rather,  experts  appear  to  be  able  to  determine  which  cues  are  the  more  useful  for  decision  making, 
and  therefore  can  select  more  appropriate  cues  than  novices  (e.g.,  Salas  et  al.,  2001).  This  finding 
will  be  discussed  more  thoroughly  in  the  training  section  (Section  5.1). 

In  cases  where  a  large  number  of  cues  must  be  integrated  such  as  in  DDM,  one  strategy  is  to  divide 
the  decisions  among  different  people.  In  such  distributed  decision  making,  different  experts  are 
responsible  for  making  sub-decisions.  This  information  is  then  passed  to  a  central  decision  maker 
who  must  integrate  these  cues,  along  with  information  about  the  validity  of  cues  and  the  skill  level 
of  the  experts.  These  central  decision  makers  are  then  responsible  for  making  the  larger  decision 
(e.g.,  Brehmer  &  Hagafors,  1986).  The  cues  in  these  scenarios  therefore  increase  in  number  as 
central  decision  makers  must  be  aware  of  these  additional  factors.  Unfortunately,  distributing  cues 
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to  different  decision  makers  is  often  not  a  practical  option.  Thus  decision  makers  have  to  develop 
ways  to  effectively  integrate  the  cues  together.  To  do  this  a  number  of  different  strategies  are 
postulated  to  be  applied. 

4.5.2  Different  Cue  Integration  Strategies 

There  are  a  variety  of  decision  making  approaches  that  can  be  applied  in  any  given  situation. 
Commonly  researched  are  compensatory  and  non-compensatory  decision  strategies.  Cue 
integration  plays  a  central  role  in  both  these  approaches.  Compensatory  strategies  involve 
integrating  all  information  to  be  used  in  the  decision.  This  means  compensatory  strategies  weigh 
information  and  opposite  values  can  cancel  each  other  out.  As  well,  the  presence  of  a  cue  with  high 
value  (weight)  can  compensate  for  an  absence  of  cues  with  low  weight  (e.g.,  evidence  for  the 
decision  can  still  be  considered  strong).  Non-compensatory  strategies  are  strategies  where  the 
“trading  off’  property  doesn’t  apply.  For  example,  every  cue  must  have  a  high  value  for  evidence 
for  the  decision  to  be  considered  strong  (Rothrock  &  Kirlik,  2003).  Thus,  non-compensatory 
strategies  involve  disregarding  certain  information  (e.g.,  in  the  “choose  the  best”  heuristic  only  the 
most  valid  cue  is  considered)  (Pachur  &  Hertwig,  2006).  Dunwoody  et  al.  (2000)  suggest  that 
neither  compensatory  nor  non-compensatory  strategies  are  inherently  better.  What  makes  one  or 
the  other  more  appropriate  has  to  do  with  the  nature  of  the  task  (e.g.,  whether  judicious  decisions 
will  have  to  be  made),  the  nature  of  the  inputs  (e.g.,  whether  cues  should  be  weighted  equally;  how 
many  cues  must  be  integrated);  the  way  information  is  given  (e.g.,  framing);  and  other  factors. 
Non-compensatory  cue  usage  strategies  seem  to  be  prevalent  in  dynamic  tasks  (Rothrock  &  Kirlik, 
2003),  which  is  reasonable  given  that  non-compensatory  strategies  often  yield  very  good  results, 
are  generally  faster,  and  require  fewer  resources  compared  to  the  more  cognitively  demanding 
compensatory  strategies  (Rothrock  &  Kirlik,  2003). 

In  other  research,  Martignon  and  Hoffrage  (2002)  reviewed  models  of  paired  comparisons  (e.g., 
choosing  one  of  two  options)  from  several  domains.  Computational  complexity  was  one  of  the  key 
variables  examined.  The  authors  report  that  a  simple  heuristic,  “take  the  best”  (TTB)  is  extremely 
simple  yet  surprisingly  robust,  and  adequately  allows  the  application  of  knowledge  to  new  data. 
This  specific  non-compensatory  strategy  has  been  studied  extensively. 

The  TTB  heuristic  can  be  used  when  a  decision  maker  must  choose  between  two  options  using  one 
criterion  (Gigerenzer  &  Goldstein,  1 999;  as  cited  in  Adams  et  al.,  2009).  When  the  cues  have  been 
learned  through  observation,  the  order  of  cues  can  be  estimated  from  the  number  of  times  the  cue 
has  predicted  the  criterion  in  the  past.  This  information  is  used  to  conduct  a  limited  search  using  a 
step-by-step  process. 

Step  1 :  Guiding  search.  Choose  the  cue  that  seems  to  be  the  most  valid.  Then  look  at  the 

values  of  each  option. 

Step  2:  Stopping  rule.  If  one  option  has  a  higher  value  than  the  other,  stop  the  search.  If 

not,  go  back  to  step  1  and  find  another  cue.  If  no  further  cue  is  found,  guess. 

Step  3:  Decision  rule.  Choose  the  object  with  the  higher  cue  value. 

In  comparing  the  TTB  heuristic,  a  non-compensatory  model,  to  the  weighted  additive  (WADD) 
strategy,  a  compensatory  model,  and  different  optimal  scenarios  were  identified  for  each.  WADD 
is  better  at  predicting  decision  making  when  information  acquisition  is  costly,  time  pressure  exists, 
or  when  information  must  be  retrieved  from  memory.  TTB,  on  the  other  hand,  is  suited  for 
predicting  when  information  acquisition  costs  are  low,  there  is  no  time  pressure  and  when 
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information  is  provided  simultaneously  (as  opposed  to  sequentially).  This  supports  the  concept  that 
people  apply  both  compensatoiy  and  non-compensatory  strategies  effectively,  changing  the  model 
applied  based  on  task  and  environmental  factors  (Ayal  &  Hochman,  2009). 

Other  TTB  versions  have  also  been  explored.  Garcia-Retamero,  Hoffrage,  and  Dieckmann  (2006) 
conducted  a  series  of  studies  that  investigated  strategies  used  in  decision  making.  The  TTB  strategy 
employs  one  reason  decision  making  (i.e.  only  focuses  on  a  single  cue  at  a  time)  and  is  non¬ 
compensatory.  These  one-reason  decision  strategies,  however,  cannot  deal  with  compound  cues 
(e.g.,  two  variables  that  together  create  an  effect  and  individually  do  not  have  the  same  effect) 
which  are  likely  to  happen  in  DDM.  TTB-Configural  is  a  strategy  that  provides  a  method  through 
which  compound  cues  are  processed.  The  underlying  theory  presents  the  idea  that  compound  cues 
(e.g.,  two  substances  appearing  in  conjunction  with  one  another)  that  have  high  validity  in 
predicting  the  criterion  are  identified  as  such.  For  example,  when  individually  these  cues  may  not 
have  any  validity  (x  has  30%  validity,  y  has  20%  validity),  but  as  a  compound  cue  they  do 
(combined  xy  has  90%  validity),  participants  appropriately  search  for  cues  that  jointly  constitute 
the  criterion  (Young,  Wasserman,  Johnson,  &  Jones,  2000;  as  cited  in  Garcia-Retamero  et  al.).  In 
studying  these  strategies,  Garcia-Retamero  et  al.  hypothesized  that  individuals  would  consider  that 
two  compounds  may  interact  with  each  other  when  they  both  act  through  a  common  causal 
mechanism  in  determining  an  effect.  They  provided  participants  with  three  different  types  of 
instruction  that  attempted  to  manipulate  participants’  causal  mental  models.  In  the  configural 
model  instructions  included  that  two  substances  acted  through  the  same  mechanism  (e.g.,  the  blood 
stream).  The  elemental  model  provided  instruction  that  stated  two  substances  acted  through 
different  mechanisms  (e.g.,  the  blood  stream  and  the  nervous  system).  Finally,  in  the  neutral  causal 
model  condition  participants  were  not  given  any  instruction  about  the  causal  mechanism. 

Findings  showed  that  when  participants  were  given  instruction  that  cues  acted  through  the  same 
causal  mechanism  the  majority  of  participants  made  decisions  best  modelled  by  the  TTB- 
Configural  strategy.  When  instructions  suggested  cues  were  acting  independently  their  decisions 
were  best  modeled  by  the  elemental  TTB.  This  was  also  the  case  for  the  neutral  causal  model 
condition.  In  an  environment  with  no  compound  cues,  no  participants  were  categorized  as  using 
TTB-Configural,  but  instead  were  modelled  best  by  the  TTB.  Thus,  the  hypothesis  was  confirmed 
that  in  situations  where  compound  cues  have  high  validity,  the  TTB-Configural  is  the  best  model  of 
the  strategy  used.  When  no  compound  cues  are  present,  participants  tended  to  revert  to  strategies 
best  modeled  by  TTB. 

Research  on  additional  alternative  TTB  strategies  has  also  been  conducted.  A  version  of  the  TTB 
strategy  used  for  classification  (TTB-C)  is  described  by  Bryant  (2007).  In  the  context  of  threat 
classification,  consider  the  case  where  there  is  an  unclassified  contact.  Using  TTB-C,  an  operator 
would  search  for  the  single  most  valid  cue  which  would  serve  as  the  basis  for  classification  (cue 
validities  would  have  to  be  determined  during  some  learning  phase).  When  the  most  valid  cue 
available  is  located,  there  is  an  assessment  about  the  appropriateness  of  the  threat  classification, 
and  then  a  determination  about  which  has  the  greater  probability  of  being  true.  If  no  valid  cue  is 
found  then  there  is  a  random  guess. 

Other  possible  ways  of  using  cues  to  perform  this  task  include: 

•  Weighted  Pros  Rule:  counts  bits  of  evidence  for  an  alternative;  examines  each  cue  value 
and  assigns  evidence  for  each  classification.  A  running  sum  is  maintained  and  after  all 
available  cues  have  been  inspected  and  weighted,  a  decision  is  made  about  the 
classification  for  which  there  is  more  evidence. 
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•  Unweighted  pros  rule:  the  same  as  the  weighted  pros  rule  except  all  cues  have  the  same 
weight. 

•  Bayesian  strategy:  conditional  probabilities  of  classifications  given  the  cue  pattern  and 
select  the  higher-probability  alternative  -  the  cue  configurations  matter. 

Bryant  (2007)  found  that  no  participants  appeared  to  use  the  Bayesian  strategy.  While  many 
strategies  were  unclassifiable,  the  majority  of  participants  appeared  to  use  either  the  TTB-C 
strategy  or  a  Pros  Rule  (most  users  of  the  Pros  Rule  used  the  Unweighted  Pros  Rule  when  the 
distinction  could  be  made  in  Experiment  2).  When  participants  were  explicitly  instructed  about  the 
Bayesian  strategy,  many  participants  used  that  strategy  when  they  had  the  time  to  do  so  (i.e.,  in  the 
low  time  pressure  condition)  showing  the  possibility  for  increased  application  of  a  Bayesian 
strategy  for  cue  integration  through  proper  training  (e.g.,  Bryant,  2005). 

Still,  TTB  is  far  from  being  accepted  in  the  literature  as  the  commonly  applied  approach.  Newell, 
Rakow,  Weston,  and  Shanks  (2004)  consider  different  strategy  types  for  cue  acquisition: 

•  Cues  which  stop  and  make  a  choice  on  the  first  discriminating  cue  that  is  found  (i.e.,  “one- 
reason”  decision  strategies: 

o  Take  the  Best  (TTB):  search  cues  in  descending  order  of  validity 
o  Draw  the  Discriminator  (DTD):  search  cues  in  descending  order  of  discrimination  rate 
o  Select  the  Successful  (STS):  search  cues  in  descending  order  of  success 

•  Strategies  that  stop  if  no  discriminating  cues  is  found  by  a  particular  point  (e.g.,  after  1, 2, 
or  3  cues) 

•  Weighted  additive  model  (multiple  linear  regression)  created  during  learning  block  of  trials 

•  Bayesian  calculation  (works  out  to  the  same  as  the  weighted  additive  model) 

The  authors  found  that  the  STS  strategy  is  highly  accurate  and  more  frugal  than  the  TTB  strategy. 
The  authors  also  note  that  participants  did  not  select  as  many  cues  as  they  probably  would  have  if 
cues  had  not  been  so  expensive  (each  cue  selected  dropped  the  expected  financial  reward  of  the 
participants). 

It  is  possible  that  participants  do  not  consistently  choose  the  same  strategies.  Factors  such  as  cue 
salience  and  other  differences  on  trials  could  have  influenced  strategy  choice  for  individual  trials. 
Individual  participants  would  also  have  experienced  different  levels  of  the  variables  early  in  the 
experiment  (e.g.,  cue  validities)  because  of  probabilistic  selection  of  items. 

More  studies  have  also  found  support  for  compensatory  strategies  over  non-compensatory  strategies. 
Newell  and  Shanks  (2004)  conduct  two  studies  that  indicate  decisions  may  be  more  compensatory  in 
nature.  They  discuss  the  non-compensatory  strategies  of  recognition  information  proposed  by 
Goldstein  and  Gigerenzer  (2002)  and  argue  that  results  from  their  studies  indicate  participants  do 
continue  to  search  above  and  beyond  recognized,  high  validity  cues.  The  experiments  were 
conducted  within  the  context  of  financial  investment.  One  may  argue  that  the  situation  and 
investment  risks  significantly  impacted  participants’  willingness  to  use  non-compensatory  strategies. 
It  is  not  expected  that  these  results  would  transfer  well  to  a  DDM  context  where  time  limitations  and 
high  uncertainty  are  prevalent,  as  this  study  was  done  in  a  static  environment. 

Prescriptive  models  suggest  that  decision  makers  should  look  at  every  cue  that  might  affect  a 
judgment.  Each  of  these  cues  must  then  be  weighed  based  on  “how  successfully  the  cue  predicts  the 
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outcome  (Shah  &  Oppenheimer,  2007,  p.  571).  This  complex  weighing  process  does  not  appear  to 
have  been  applied  in  all  the  studies  summarized  above,  where  prescriptive  approaches  have  been 
abandoned  for  simpler  non-compensatory  valuations.  While  this  does  seem  to  be  an  effective 
approach  in  the  environments,  and  also  appears  to  serve  decision  makers  well  in  day-to-day  reasoning 
(Klein  &  Crandall,  1996),  the  question  remains  about  whether  or  not  this  is  an  effective  approach  in 
DDM  situations.  Explicitly  exploring  these  approaches  in  DDM  should  prove  interesting. 

4.5.3  Making  Inferences 

Making  inferences,  or  connections  between  ideas,  is  one  function  of  working  memory.  One  such 
inference  that  is  important  in  cue  integration  is  making  inferences  about  cues  and  criteria;  that  is, 
predicting  the  future  state  of  the  DDM  environment  based  on  its  current  state  (including  the  decision 
maker’s  own  actions).  Challenges  in  making  inferences  about  the  relationships  between  cues  and 
criteria  are  pervasive  in  DDM  contexts.  Understanding  what  cues  should  be  used  when  making 
decisions  is  generally  a  function  of  understanding  cue  validity;  that  is,  how  predictive  of  an  outcome 
is  a  given  cue?  As  well,  one  of  the  most  important  types  of  cue-criterion  relationship  that  must  be 
perceived  and  understood  in  order  to  improve  decision-making  performance  is  the  relationship 
between  the  action  of  the  decision  maker  and  the  results  of  those  actions  (i.e.,  feedback).  In  non- 
DDM  environments,  feedback  about  performance  or  other  variables  is  often  given  separately  after  a 
decision  and  acts  only  as  an  output  of  rather  than  an  input  to  decision  making  (see  Section  4.1.1  for  a 
more  detailed  discussion  about  why  we  are  treating  feedback  as  a  cue  in  DDM  environments,  and 
Section  5.3.3  for  a  discussion  of  providing  feedback  during  DDM  training).  Because  feedback  is  such 
an  important  type  of  cue  it  is  dealt  with  in  its  own  section  (Section  4.7). 

4.5.3.1  Cue  Validity 

Determining  cue  validity  (i.e.,  how  much  weight  to  assign  to  each  cue)  is  a  crucial  part  of  cue 
integration.  Newell  and  Shanks  (2004)  conducted  two  experiments  that  investigated  cue  validity  in 
decision  making.  It  was  found  that  the  validity  of  the  cues  was  the  primary  factor  in  determining  if 
it  would  be  used  as  the  main  information  in  the  decision-making  process. 

This  study  demonstrated  that,  when  the  validity  of  a  cue  is  kept  consistent  across  trials,  over  time 
participants  are  able  to  identify  its  validity  and  use  the  information  in  making  accurate  decisions. 

This  evidence  is  presented  in  the  context  of  simple  decision  making  where  the  dynamic  of  the 
situation  does  not  change  the  cue  validity  across  trials.  One  would  expect  that  the  uncertainty 
associated  with  this  type  of  change  will  have  a  significant  impact  on  the  ability  of  people  to  grasp 
the  validity  of  a  cue  in  a  particular  situation.  This  study  indicates  that  in  order  for  cue  validity  to  be 
understood,  multiple  trials  are  required  where  the  cue  validity  is  kept  constant.  This  is  evidence  for 
the  fact  that  expertise  in  decision  making  might  be  a  function  of  learning  cue-criterion 
relationships,  and  that  training  such  relationships  could  improve  DDM  performance.  The  specific 
application  of  this  in  military  DDM  environments  should  be  more  explicitly  explored. 

4.5.3.2  Mental  Models 

Another  way  in  which  inferences  between  cues  are  made  is  through  the  formation  of  mental 
models.  Mental  models  are  created,  accessed  and  manipulated  in  working  memory  and  cues  and 
cue  integration  play  a  central  role  in  this  process.  Thus,  a  brief  discussion  on  the  use  of  cues  in 
mental  models  is  required.  Cohen  et  al.  (1999)  argued  that  optimal  performance  in  DDM  contexts 
rely  upon  appropriate  mental  models  about  organizational  purpose,  own  and  others’  intent,  degrees 
of  initiative,  team  member  reliability,  and  action  sequences.  To  make  the  best  use  of  this 
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information,  the  ability  to  think  critically  about  these  mental  models  and  apply  them  to  novel 
situations  is  also  required.  According  to  the  authors,  a  fundamental  skill  is  related  to  separating 
grounds  (what  is  given)  from  conclusions  (what  is  inferred).  The  relationship  between  grounds  and 
conclusions  is  generally  not  100%  certain,  so  this  is  a  way  of  identifying  assumptions  that  may 
have  to  be  tested.  This  can  be  understood  as  organizing  and  understanding  how  different  cues  (e.g., 
ground  information  and  conclusions)  are  integrated  within  a  conceptual  system  used  in  DDM. 

From  a  mental  models’  perspective  of  understanding  situation  awareness,  Endsley  (1995,  p.  43) 
describes  a  situation  model  as  “a  schema  depicting  the  current  state  of  the  mental  model  of  the 
system”.  A  situational  model  is  a  dynamic  representation  of  a  person’s  knowledge  and  understanding 
of  the  present  state  of  a  system  (Endsley,  2000;  as  cited  in  Brown  et  al.,  2009).  It  may  incorporate  the 
value  of  different  cues  (e.g.,  level  of  fuel  gauge);  an  integration  of  the  cues  to  understand  the 
dynamics  of  the  system  (e.g.,  rate  of  change)  developed  from  changes  in  cues  from  the  situation 
model  over  time;  higher  level  comprehension  (e.g.,  understanding  the  state  of  the  system  in  a  Gestalt 
form  in  relation  to  human  goals);  and  projections  of  future  states.  A  situation  model  may  also  assign 
well  understood  situations  to  prototypical  classifications  of  the  system  state  (Brown  et  al.) 

As  developing  mental  models  is  mainly  cognitive,  methods  on  how  this  is  accomplished  can  only 
be  theorized.  Of  importance  to  our  discussion  are  the  three  levels  that  make  up  situation  models 
(Endsley  et  al,  1995).  The  first  level  is  perception  of  cues.  One  needs  basic  perception  of  important 
information  in  order  to  form  a  correct  picture  of  the  situation.  The  second  level  is  comprehension, 
which  refers  to  one’s  ability  to  integrate  multiple  pieces  of  information  and  determine  their 
relevance  to  one’s  goals.  The  final  level  is  projection,  which  refers  to  the  ability  to  project  future 
events  from  current  events  and  dynamics.  It  is  specifically  the  second  level  that  identifies  the 
importance  of  cue  integration  within  the  development  of  a  mental  model.  The  relevance  lies  in  the 
connection  that  mental  models  are  identified  as  a  key  component  of  decision  making  in  DDM 
situations  (Brown  et  al.) 

Another  perspective  comes  from  Richardson,  Andersen,  Maxwell,  and  Stewart  (1994;  as  cited  in 
Brown  et  al.,  2009)  who  examine  the  concept  of  mental  models  in  the  field  of  system  dynamics. 
Incorporating  ideas  from  the  Brunswickean  lens  model  into  a  feedback  theory  model,  a 
representation  of  mental  models  is  developed.  Within  the  Brunswickean  lens  model,  the  “perceived 
state”  is  a  complex  process  involving  true  descriptors  of  the  state  of  the  system,  cues  and 
measurements  derived  from  the  descriptors,  and  subjective  cues.  Subjective  cues  are  a  combination 
of  objective  cues  unconsciously  assembled  by  the  observer  (Richardson  et  al,  1994;  as  cited  in 
Brown  et  al,  2009).  They  are  the  result  of  scanning  a  field  for  all  possible  cues,  selecting 
combinations  of  cues  to  attend  to,  and  gauging  the  cues.  These  cues  are  then  subjectively 
interpreted  in  light  of  experience,  memory,  and  perceptions.  Assessments  of  the  current  system 
state  and  predictions  for  future  states  are  made  from  these  subjective  cues. 

Within  the  model  developed  by  Richardson  et  al.  (1994;  described  by  Brown  et  al.,  2009),  an 
observer  scans  the  system,  selects  and  interprets  cues  from  it,  and  subjectively  interprets  them  as  a 
basis  for  assessing  and  predicting  the  state  of  the  system.  Assessments  and  predictions  are  based  on 
the  observer’s  cognitive  model  of  the  system.  Comparisons  of  the  desired  state  and  the  current  state 
result  in  the  selection  of  strategies,  tactics,  and  policy  levels.  Also  included  in  this  model  were 
mechanisms  for  learning  as  state  that  learning  is  the  mix  of  processes  that  allow  people  to  change 
their  mental  models. 

Brown  et  al.’s  (2009)  review  of  mental  models  within  DDM  understands  the  integration  of  cues 
within  a  dynamic  environment  as  effortful.  This  is  reflective  of  the  requirement  for  the  decision 
maker  to  constantly  integrate  new  cues  to  update  their  mental  model  and  decipher  ambiguous  cues  in 
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establishing  coherence.  It  is  proposed  that  establishing  a  new  mental  model  requires  much  more 
effort  and  is  more  complex  relative  to  simply  adding  new  information  into  a  model.  The  effort  then 
required  to  integrate  cues  may  vary  depending  on  previously  established  mental  models  which  may 
come  from  experience  within  similar  problem  solving  situations.  In  addition,  Brehmer  and  Allard 
(1991)  discuss  complexity  (as  is  found  in  DDM)  as  having  a  relevant  effect  on  the  amount  of 
information  that  must  be  integrated.  Additional  challenges  within  DDM  (e.g.,  time  constraints) 
would  be  expected  to  also  have  a  significant  impact  on  the  use  of  cues  in  developing  mental  models. 

Although  there  is  a  vast  literature  on  mental  models,  very  little  literature  exists  that  provides  a 
description  of  the  cognitive  processes  involved  when  mental  models  are  formed  (Brown  et  al., 

2009).  The  role  of  cue  integration  within  the  development  of  mental  models,  at  this  point,  can  only 
be  hypothesized.  The  ideas  reviewed  here  however,  suggest  that  mental  models  are  built  from  the 
perception  and  integration  of  cues  such  as  knowledge  required  to  pursue  a  goal  and  the  data 
perceived  from  the  environment  (Besnard,  Greathead,  &  Baxter,  2004;  as  cited  in  Brown  et  al.) 
Mental  models  are  formed  after  an  observer  scans  the  environment  of  interest,  selects  cues  from  this 
environment,  and  subjectively  interprets  the  cues  (Richardson  et  al.,  1994,  as  cited  in  Brown  et  al.). 
There  is  evidence  that  suggests  cue  integration  plays  a  key  role  in  DDM  through  mental  models.  The 
resources  and  cognitive  effort  required  for  the  process  of  cue  integration  may  depend  on  the  state  of 
established  mental  models  as  well  as  the  complexity  of  the  problem.  For  a  more  detailed  discussion 
about  the  role  of  mental  models  in  dynamic  decision  making  refer  to  Brown  et  al. 

4.6  Long  Term  Memory  and  Cue  Integration  in  DDM 

Long-term  memory  is  a  large-capacity  storage  system  which  can  maintain  information  for  a  long 
duration.  Recall  from  the  information  processing  model  presented  earlier  (Figure  1)  that 
information  may  move  from  working  memory  into  long-term  memory  if  certain  conditions  are 
present,  for  example  if  information  is  paid  attention  to.  Likewise,  information  may  move  from 
long-term  to  working  memory  (i.e.,  information  may  be  remembered). 

Note  that  long-term  memory  and  learning  are  closely  linked.  Information  has  to  be  stored  in  long¬ 
term  memory  before  it  can  be  used  to  influence  later  behaviour,  and  so  anything  that  provides  a 
challenge  to  long-term  memory  performance  is  also  likely  to  be  a  challenge  for  learning  and  training. 

The  research  reviewed  for  this  project  identified  challenges  related  to  long-term  memory,  including: 

•  Forgetting 

•  Biased  recall 

•  Resources  required  to  purposefully  recall  information 

•  Unintentional  recall  of  information  which  influences  decisions  in  ways  not  intended  (or 
even  noticed  by  the  decision  maker) 

Although  long-term  memory  has  a  large  capacity  and  stores  information  for  a  long  duration,  getting 
information  out  of  long-term  memory  and  into  working  memory  is  often  an  effortful  procedure 
(i.e.,  it  requires  cognitive  resources)  and  often  fails  (i.e.,  we  forget).  Important  cues  can,  therefore, 
fail  to  be  included  in  decision  making  and  learning  effective  decision-making  strategies  can  be 
difficult  (e.g.,  Bakken,  2008;  Bonifay,  2000). 

It  has  been  found  that  the  manner  in  which  information  is  retrieved  from  long-term  memory  can  be 
biased  by  the  order  in  which  cues  were  learned,  and  how  long  ago  they  were  learned.  For  example. 
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Stout,  Amundson,  and  Miller  (2005)  Investigated  potential  changes  in  trial  order  effects  as  a 
function  of  retention  interval  in  a  predictive  judgment  task.  Specifically,  the  authors  wanted  to 
know  whether  a  long  retention  interval  before  a  judgment  task  would  affect  the  cue-outcome 
relationships  that  are  used  to  make  a  judgment.  In  this  experiment,  the  investigators  used  cue- 
outcome  relationships  similar  to  those  from  Pavlovian  experiments  (i.e.,  non-reinforced  and 
reinforced  trials)  and  changed  the  order  in  which  they  were  presented  (i.e.  non-reinforced  both 
preceding  and  following  reinforced  trials).  Participants  performed  the  judgment  task  and  were 
tested  either  immediately  or  48  hours  after  the  experiment.  Results  showed  that  trial  order  and 
retention  interval  interacted  to  influence  the  predictive  ratings.  Specifically,  participants  tested  right 
after  the  experiment  gave  predictive  ratings  that  reflected  most  recently  experienced  trial  types 
thereby  showing  a  recency  effect.  On  the  other  hand,  following  48  hours,  predictive  ratings  were 
congruent  with  first-experienced  trial  types  thereby  showing  a  primacy  effect.  Overall,  the  results 
show  that  people  are  influenced  by  the  order  of  events  when  making  future  predictions  (Stout  et  al., 
2005)  It  is  likely  that  the  demonstrated  switch  between  recency  and  primacy  effects  over  time 
would  still  be  present  in  a  DDM  environment,  and  when  decision  making  takes  place  relative  to 
learning  could  impact  how  and  what  cues  are  used.  This  could  impact  the  consistency  of  decision 
making  in  an  extended  DDM  task  (e.g.,  for  more  than  48  hours  such  as  in  many  military 
operations)  if  different  information  is  used  at  different  times. 

Long-term  memory  has  a  large  impact  on  other  processes  through  the  influence  of  experience.  This 
is  often  unnoticed  by  decision  makers  (i.e.,  it  is  unconscious),  and  so  decision  makers  are  left 
unaware  of  factors  that  influence  decision  making.  This  means  that  consciously  influencing  the 
decision-making  process  can  be  difficult,  and  learning  and  teaching  decision  making  strategies  can 
be  challenging. 

The  literature  we  reviewed  related  to  long-term  memory  and  cue  integration  did  not  involve  DDM 
tasks.  However,  it  is  reasonable  to  suggest  that  similar  effects  related  to  forgetting,  biased  recall, 
difficulties  in  deliberate  recall,  and  unintentional  recall  would  be  relevant  in  DDM.  If  anything, 
these  challenges  are  likely  to  play  an  even  bigger  role  in  DDM  than  in  simple  decision  making 
tasks,  as  DDM  is  likely  to  be  more  cognitively  demanding. 

Although  deliberately  retrieving  information  from  long-term  memory  can  be  effortful,  often 
information  is  retrieved  without  any  desire  by  the  decision  maker  and  can  interfere  with  decision 
making  (e.g.,  intrusive  thoughts).  This  means  that  information  which  the  decision  maker  does  not 
want  to  be  a  factor  in  the  decision  could  influence  the  decision  regardless.  One  type  of  knowledge 
that  is  stored  in  long-term  memory  that  has  been  examined  is  causal  knowledge. 

4.6.1  Causal  Knowledge 

It  is  important  to  remember  that  cue  integration  in  DDM  takes  place  in  context;  not  only  the 
context  provided  by  the  specific  task  and  circumstance  in  which  the  task  is  taking  place,  but  also  a 
context  provided  by  the  background  knowledge  of  the  decision  maker.  This  background 
knowledge,  stored  in  the  long-term  memory  of  the  decision  maker,  can  influence  cue  integration. 
One  type  of  background  knowledge  which  has  been  examined  in  the  context  of  decision  making  is 
causal  knowledge;  that  is,  the  role  of  causes  and  effects  and  how  they  influence  one  another. 

Causal  knowledge  has  been  shown  to  influence  the  type  and  amount  of  cues  which  are  sought,  how 
cues  are  interpreted,  how  easily  relationships  between  cues  and  criteria  are  remembered,  and  the 
type  of  cue  integration  strategy  that  is  used.  Essentially,  causal  information  acts  as  a  “meta-cue”  or 
a  cue  about  cues  to  help  determine  what  is  helpful  (Garcia-Retamero  et  al.,  2006).  An  important 
question  is  whether  the  influence  of  causal  knowledge  is  automatic  or  controlled.  This  will  have 
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implications  for  whether  causal  knowledge  could  be  accessible  to  consciousness  and  modifiable 
through  training. 

Causal  knowledge  has  been  found  to  influence  the  perceived  strength  between  cues  and  outcomes. 
According  to  Garcia-Retamero  et  al.  (2006),  people  intuitively  believe  that  cues  with  causal 
relationships  with  decisions  being  made  have  stronger  relationships  with  potential  outcomes  (i.e., 
they  are  more  valid),  and  so  causal  cues  are  more  likely  to  be  selected  than  non-causal  cues, 
although  more  valid  non-causal  cues  will  be  selected  over  causal  cues  which  are  much  less  valid. 
This  indicates  both  the  relationships  between  cues  and  decisions  as  well  as  overall  actual  cue 
validity  are  used  when  selecting  cues. 

Garcia-Retamero  et  al.  (2006)  found  that  causal  knowledge  can  influence  whether  participants 
would  search  for  more  information,  as  well  as  which  cue  integration  strategy  they  choose.  The 
authors  examined  how  participants  deal  with  compound  cues.  Participants  were  given  a  task  that 
required  them  to  make  a  decision  about  what  substance  patients  ingested  based  on  symptomatic 
information.  Classification  of  participants  according  to  the  strategies  they  used  was  based  on  the 
number  of  cues  used  and  decision  outcomes.  Results  showed  that  when  participants  were  given 
instruction  that  cues  acted  through  the  same  causal  mechanism,  the  majority  of  participants  made 
decisions  best  modelled  by  the  TTB-Configural  heuristic  (searching  for  a  particular  set  of  cues). 
When  instructions  suggested  cues  were  acting  independently  their  decisions  were  best  modeled  by 
the  elemental  TTB  heuristic  (searching  for  only  the  most  valid  cue).  Participants  given  causal 
mechanism  instructions  sought  out  more  cues  during  the  experiment  as  they  may  have  been  more 
uncomfortable  making  the  decision  due  to  their  original  “mental  model”  that  did  not  prove  useful 
in  an  environment  with  no  compound  cues.  This  study  demonstrates  that  confidence  in  decision 
making  and  comfort  level  with  the  amount  and  quality  of  the  information,  seems  to  be  a  factor  in 
determining  whether  to  search  for  more  information  in  attempts  to  use  a  more  analytic  strategy,  or 
simply  to  use  fast  and  frugal  heuristics  with  minimal  information.  When  cues  have  a  more  direct, 
causal  relationship  to  the  criterion,  the  strategies  used  imply  a  type  of  “mental  model”  (what  the 
authors  define  as  a  mental  representation  of  the  problem  solving  strategy  structure)  that  gathers  and 
processes  individual  pieces  of  information  as  well  as  compound  pieces  of  information 
(combinations  of  cues)  rather  than  individual  cues.  If  the  same  type  of  finding  applies  to  DDM, 
then  understanding  how  decision  makers  represent  the  situation  (e.g.,  the  extent  to  which  they  rely 
on  mental  models)  would  be  important  for  predicting  how  they  integrate  cues,  and  whether  they 
would  seek  individual  cues  or  cue  combinations  with  which  to  make  their  decisions. 

Causal  knowledge  can  also  influence  how  easily  cue-criterion  relationships  can  be  learned  which 
seems  to  interact  with  the  type  of  task  being  performed.  For  example,  Cobos,  Lopez,  and  Luque 
(2007)  investigated  the  effect  of  causal  reasoning  on  whether  or  not  there  is  interference  when 
learning  that  multiple  cues  are  related  to  the  same  outcome.  The  problem  in  the  real  world  is  that 
causal  relationships  have  to  be  inferred,  and  an  assessment  of  when  they  are  and  when  they  are  not 
applicable  must  be  made.  The  authors  use  retroactive  interference  to  investigate  this  process. 
Retroactive  interference  occurs  when  learning  something  new  interferes  with  something  already 
learned,  for  example  when  you  get  a  new  phone  number  and  then  can’t  remember  the  one  you  used 
to  have.  In  this  context,  participants  are  learning  cue-outcome  relationships.  Once  a  cue-outcome 
relationship  is  learned,  learning  a  new  relationship  between  a  different  cue  and  the  same  outcome 
interferes  with  the  previous  learning.  The  initial  cue-outcome  is  not  “unlearned”  as  proper 
contextual  effects  can  reinstate  the  memoiy.  Rather,  the  initial  cue-outcome  is  harder  to  retrieve 
without  these  contextual  “hints”.  Cobos,  Lopez,  and  Luque  interpreted  this  finding  to  mean  that  the 
first  relationship  is  deemed  to  not  be  valid  in  the  same  situations  as  the  second  relationship.  Note 
that  in  the  situation  described  here  there  is  no  reason  to  actually  assume  that  the  two  relationships 
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cannot  coexist  (as  there  would  be  if  the  same  cue  predicted  two  different  outcomes).  This  finding 
would  be  particularly  relevant  in  DDM  contexts  in  which  relationships  between  cues  and  outcomes 
change  over  time.  More  research  needs  to  be  done  to  understand  under  what  conditions  cue- 
criterion  relationship  changes  are  actually  detected,  and  under  what  conditions  old  cue-criterion 
relationships  are  deemed  to  be  relevant  again. 

Cobos,  Lopez,  and  Luque  (2007)  suggested  that  there  are  differences  in  cue  interpretation  in 
prediction  tasks  (e.g.,  see  how  many  clouds  are  in  the  sky  and  predict  whether  it  will  rain)  where 
cues  are  causes  (i.e.,  clouds),  and  in  diagnostic  tasks  (e.g.,  need  to  determine  the  cause  of  a  rash) 
where  cues  are  effects  (i.e.,  the  rash).  The  authors  found  an  initial  retroactive  interference  effect  in 
the  diagnostic  task  in  the  same-outcome  group  compared  to  the  different-outcome  group  (i.e., 
performance  was  worse  when  different  cues  were  related  to  the  same  outcomes  in  the  two  phases). 
They  also  found  the  interference  effect  was  larger  for  the  diagnostic  than  predictive  condition. 
Finally,  findings  suggest  there  is  no  significant  interference  effect  in  the  predictive  condition.  Thus, 
the  type  of  task  (predictive  versus  diagnostic)  influences  how  easily  causal  links  between  cues  and 
criteria  can  be  learned. 

Vandorpe,  Houwer,  and  Beckers  (2007)  attempted  to  determine  the  extent  to  which  the  use  of 
causal  knowledge  is  automatic  or  controlled  in  the  context  of  causal  learning  tasks.  In  causal 
learning  tasks,  participants  are  presented  with  combinations  of  cues  that  are  potential  causes  of  an 
outcome.  After  learning,  participants  are  asked  to  assess  for  each  cue  separately  the  probability  that 
it  causes  the  outcome.  The  cue  competition  effect  refers  to  the  finding  that  a  causal  judgment  about 
the  likelihood  that  a  target  cue  will  produce  a  particular  outcome  is  not  only  determined  by  the 
probability  that  that  target  cue  predicts  the  outcome,  but  also  by  the  probability  that  the  outcome 
was  caused  by  another  cue  that  co-occurred  with  the  target  cue  during  learning.  Typical  cue- 
competition  effects  include  the  following: 

•  Forward  blocking:  presenting  cues  where  A+  trials  (cue  A  presented  alone  which  predicts 
the  outcome)  are  followed  by  AX+  trials  (cue  A  and  cue  X  are  presented  and  together  they 
predict  the  outcome)  -  causal  judgements  for  cue  X  will  be  lower  than  when  only  AX+ 
trials  are  presented 

•  Backward  blocking:  AX+  trials  are  followed  by  A+  trials  -  causal  judgments  for  cue  X  will 
be  lower  than  when  only  AX+  trials  are  presented 

•  Reduced  overshadowing:  B-  trials  (cue  B  is  presented  and  it  does  not  predict  the  outcome) 
are  followed  by  BY+  trials  -  the  causal  judgment  for  cue  Y  will  be  higher  than  when  no  B- 
trials  precede  the  BY+  trials 

•  Release  from  overshadowing:  same  effect  as  reduced  overshadowing  but  trials  occur  in 
opposite  order  (BY+  trials  then  B-  trials) 

Cue-competition  effects  are  thought  to  result  from  inferences  being  made,  and  manipulations  of 
outcome  maximality  (when  the  outcome  occurs  with  the  highest  intensity  possible)  and  additivity 
(when  two  associated  cues  occurring  together  mean  that  the  outcome  was  more  intense  than  when 
either  cue  occurred  alone)  influence  the  conditions  under  which  inferences  will  be  valid.  Thus, 
these  manipulations  should  have  an  influence  on  the  blocking  effects  to  the  extent  that  these  are 
controlled  processes.  However,  these  manipulations  are  not  related  to  the  overshadowing  effects. 
Findings  have  demonstrated  that  maximality  and  additivity  influence  blocking  effects  but  not 
overshadowing  effects;  however,  these  have  heretofore  used  very  few  cues  and  events.  Findings 
largely  support  the  theory  that  higher-order  reasoning  plays  a  prominent  role  in  human  causal 
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learning.  However,  findings  were  not  as  strong  as  expected,  indicating  that  some  of  the  effects  of 
causal  reasoning  may  occur  automatically. 

In  summary,  causal  knowledge  is  used  as  a  context  for  understanding  the  relationships  between  cues 
and  criteria.  Causal  knowledge  increases  the  perceived  validity  of  cue-criterion  relationships, 
perhaps  through  the  mechanism  of  mental  models  which  are  used  to  explain  relationships  between 
environmental  variables.  When  cue-criterion  relationships  are  consistent  with  likely  causal 
relationships,  they  appear  to  be  easier  to  remember.  In  addition,  whether  causal  links  between  cues 
and  criteria  are  perceived  or  not  may  influence  the  cue  integration  strategy  which  is  chosen.  The  use 
of  causal  knowledge  may  be  partial  ly  controlled,  but  there  is  also  evidence  that  the  use  of  causal 
knowledge  is  not  under  complete  conscious  control;  thus  causal  reasoning  may  play  an  important 
role  in  cue  integration  and  may  not  be  amenable  to  changes  during  training.  Components  of  long¬ 
term  memory  such  as  experience  (i.e.,  learning)  and  causal  knowledge  playing  a  role  in  selecting  and 
interpreting  cues  will  likely  be  more  important  for  military  decision  makers,  as  military  scenarios 
typically  involve  complex  environments  in  which  cues  may  have  to  be  actively  sought  and  may  be 
more  ambiguous  than  in  the  types  of  decision  making  tasks  used  in  the  experiments  reviewed  here. 

4.7  Feedback  and  Cue  Integration  in  DDM 

Feedback  is  a  very  important  type  of  cue  to  consider  in  cue  integration.  In  the  information 
processing  model  discussed  in  Section  4.2  (see  Figure  2),  feedback  is  represented  by  inputs  to  the 
sensory  processing  system  from  the  environment.  In  an  examination  of  the  model,  it  can  be  seen 
that  the  actions  of  the  decision  maker  (i.e.,  response  execution)  influences  the  environment. 

Feedback  is  generally  interpreted  as  information  relating  to  the  impact  that  the  decision-maker’s 
actions  have  on  the  environment.  It  is  important  for  understanding  the  current  situation  (which  has 
been  affected  by  the  previous  actions  of  the  decision  maker)  as  well  as  learning  the  relationship 
between  the  actions  of  the  decision  maker  and  outcomes.  It  should  be  noted  that  in  a  typical  DDM 
environment,  information  feeding  into  the  sensoiy  processing  system  from  the  environment  (called 
“environmental  feedback”  in  the  model)  will  also  change  due  to  factors  other  than  actions  of  the 
decision  maker.  In  this  section,  we  will  consider  the  types  of  feedback  that  are  relevant  to  DDM. 

Feedback  is  critical  for  learning  the  relationship  between  actions  and  their  consequences.  However, 
in  DDM  feedback  delays  as  well  as  ambiguous  feedback  can  complicate  this  learning  process.  In 
addition  to  the  actual  lack  of  unambiguous  feedback  and  feedback  delays  in  DDM,  it  is  also  true 
that  decision  makers  do  not  seem  to  make  ideal  use  of  the  feedback  that  is  available.  Bois  (2002) 
has  discussed  the  fact  that,  with  “perfect”  feedback,  people  should  be  able  to  perform  perfectly 
(after  some  amount  of  learning).  However,  most  often,  the  availability  of  “perfect  information”  is 
an  assumption,  and  how  participants  interpret  the  information  (i.e.,  in  a  manner  consistent  with  the 
experimenter’s  interpretation)  is  not  investigated.  In  fact,  there  is  evidence  that,  when  the 
understanding  of  participants  is  actually  tested,  although  “perfect”  information  has  been  presented, 
participants  do  not  actually  have  a  perfect  understanding. 

There  is  other  evidence  that  understanding  and  incorporating  feedback  in  a  DDM  context  may  not 
be  done  optimally.  For  example,  Sterman  (1989)  found  that  in  DDM  situations,  subjects  showed  a 
general  lack  of  consideration  of  feedback,  including  misperceptions  of  time  delays,  misperceptions 
of  feedback  from  decision  to  the  environment  and  a  tendency  to  think  in  causal  single  strand  series 
(i.e.,  people  oversimplify  the  relationship  between  actions  and  outcomes).  Further,  Gonzales  et  al. 
(2003)  found  that  when  modelling  behaviour,  models  without  feedback  were  more  accurate  at 
representing  human  performance  that  those  with  feedback.  Collectively,  these  studies  suggest  that 
feedback  may  not  be  readily  incorporated  as  a  cue  in  dynamic  decision  making. _ 
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Another  important  factor  to  consider  regarding  feedback  in  DDM  is  whether  there  are  different 
types  of  feedback  and  whether  these  have  different  effects  on  DDM  performance.  Balzer,  Sulsky, 
Hammer,  and  Sumner  (1992)  investigated  the  role  that  different  kinds  of  feedback  play  in 
improving  performance  in  multiple-cue  probability  learning  (MCPL)  tasks.  The  components  of 
cognitive  feedback  were  proposed  to  be: 

•  Task  Information  (TI):  information  about  the  task  system  (e.g.,  relations  between  cues  and 
criteria) 

•  Cognitive  Information  (Cl):  information  about  the  subject’s  cognitive  system  (e.g., 
information  about  judgment  consistency) 

•  Functional  Validity  Information  (FVI):  information  about  the  relationship  of  the  task 
system  to  the  cognitive  system  (e.g.,  correlation  between  judgments  and  the  actual  value  of 
the  criterion) 

In  simple  terms,  TI  represents  how  they  should  be  performing  the  judgment  task,  Cl  represents  how 
they  are  doing  the  judgment  task,  and  FVI  represents  the  “success”  of  the  judgment  strategy. 

Dividing  feedback  into  types  like  this  is  useful  as  it  helps  to  uncover  which  types  of  information 
actually  can  be  used  to  improve  performance,  and  also  allows  an  examination  of  what  happens 
when  multiple  types  of  feedback  are  combined. 

Balzer  et  al.’s  (1992)  findings  supported  earlier  research,  which  indicated  that  TI  plays  the  most 
important  role  in  improving  performance.  TI  feedback  alone  was  superior  to  the  no-feedback 
condition.  Unless  TI  feedback  was  received,  performance  did  not  significantly  differ  from 
conditions  in  which  no  feedback  at  all  was  received.  The  addition  of  either  type  of  feedback  to  the 
TI  feedback  (i.e.,  Cl  and/or  FVI  feedback  plus  TI  feedback)  did  not  significantly  improve 
performance  above  TI  feedback  alone. 

Essentially,  there  are  different  types  of  feedback,  and  information  about  the  task  itself  (i.e., 
relationships  between  variables)  appears  to  have  the  most  beneficial  impact  on  performance.  An 
important  consideration  when  attempting  to  apply  these  findings  to  DDM  is  that  it  might  be  very 
difficult  to  determine  what  TI  is  and  how  it  may  change.  As  well,  it  seems  a  bit  odd  to  not  receive 
any  feedback  about  the  success  of  one’s  performance  (i.e.,  FVI).  In  essence  TI  appears  to  be 
modelling  (i.e.,  showing  how  something  should  be  done)  rather  than  a  more  traditional 
understanding  of  feedback  (i.e.,  giving  information  about  one’s  own  performance). 

Other  findings  indicate  that  providing  feedback  about  relationships  between  variables  improves 
performance  more  than  feedback  about  performance  itself.  According  to  Karelaia  and  Hogarth 
(2008),  people  learn  best  from  feedback  that  instructs  them  about  the  characteristics  of  the  tasks  they 
face.  Haynie  and  Shepherd  (2007)  described  an  experiment  to  determine  the  effects  of  cognitive 
feedback  (i.e.,  instruction  about  the  underlying  relationships  between  variables;  this  corresponds  to 
task  information  rather  than  cognitive  information  as  described  by  Balzer  et  al.,  1992)  versus 
performance  feedback  (i.e.,  feedback  about  overall  decision  quality  only;  this  corresponds  to  Balzer  et 
al.’s  functional  validity  information)  on  performance  in  a  dynamic  environment.  They  found  that 
cognitive  feedback  was  more  effective  in  improving  performance  than  performance  feedback. 

In  much  of  the  training  research,  feedback  on  performance  is  often  provided  to  participants;  however, 
there  is  some  evidence  that  participants  may  be  able  to  make  effective  use  of  occasional  feedback 
when  they  seek  it  out.  Klayman  (1988)  found  that  gradual  learning  with  feedback  that  was  sought  out 
by  the  participant  on  a  limited  number  of  trials  facilitated  the  learning  of  cue-criterion  relationships, 
although  there  appeared  to  be  no  direct  comparison  with  a  constant-feedback  condition. 
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Individual  differences  also  appear  to  affect  the  degree  to  which  feedback  is  used  to  improve 
performance.  Haynie  and  Shepherd  (2007)  proposed  that  metacognitive  awareness  would  affect  the 
ability  of  a  decision  maker  to  make  use  of  feedback.  They  found  that  metacognitive  awareness 
moderated  the  relationship  between  feedback  and  performance,  with  participants  with  more 
metacognitive  awareness  benefitting  more  from  cognitive  feedback  (i.e.,  feedback  about  the 
variable  relationships). 

According  to  Juslin,  Olsson  et  al.  (2003),  the  type  of  feedback  provided  can  affect  how  a  task  is 
performed.  They  propose  that  mental  cue  abstraction  (e.g.,  information  from  cues  are  added,  and 
optimal  weights  are  also  remembered  and  used  in  making  the  judgment)  as  a  cue  integration 
strategy  is  favoured  when  a  continuous  criterion  is  used,  whereas  a  binary  criterion  tends  to  be 
interpreted  via  exemplar  memory  (e.g.,  and  example  is  remembered  and  the  current  pattern  is 
compared  to  it,  and  a  match  or  no  match  decision  is  made).  The  authors  indicate  that  this  might  be 
due  to  more  informative  feedback  in  the  continuous  judgment  paradigm.  The  authors  propose  that 
if  feedback  is  sufficient  to  infer  relations  between  cues  and  the  criterion  used  for  judgment,  then 
multiple  cue  abstraction  will  be  favoured.  With  insufficient  feedback,  rules  cannot  be  inferred  so 
exemplar  memory  is  the  most  appropriate  guide.  From  the  results  of  the  experiment  by  Juslin, 
Olsson  et  al.,  it  appears  that  the  level  of  feedback  in  different  important  tasks  should  be  examined 
to  determine  whether  cue  abstraction  is  a  viable  method  of  making  decisions.  Although  cue 
abstraction  might  be  a  preferred  method  of  making  decisions,  there  may  not  be  adequate 
information  for  it  to  be  used  effectively.  As  well,  as  there  are  individual  differences  in  decision¬ 
making  preference  between  the  two  methods,  perhaps  people  should  be  trained  in  their  natural 
style,  or  they  may  otherwise  not  perform  optimally,  especially  under  stress  and  time  pressure. 

Juslin,  Olsson  et  al.  (2003)  also  provided  evidence  that  the  type  of  feedback  provided  can  affect 
which  cue  integration  strategy  is  used.  The  authors  compared  three  different  possible  models  of 
how  people  categorize: 

1 .  Cue  abstraction  model:  abstraction  of  cue-criterion  relations  which  are  then  used  to  make 
judgments 

2.  Lexicographic  heuristic:  uses  only  the  most  valid  cue  (that  can  discriminate  between 
options;  basically,  the  same  as  the  “take  the  best”  heuristic  discussed  elsewhere) 

3.  Exemplar-based  model:  retrieve  entire  exemplars  and  use  those  to  make  categorization 
judgments  (this  seems  to  basically  be  pattern  recognition) 

The  authors  found  that  exemplar  memory  seems  to  be  preferred  when  feedback  is  poor,  but  when 
feedback  is  adequate  for  people  to  extract  cue-criterion  relations  they  tend  to  do  so. 

Another  reported  effect  of  feedback  was  reported  by  Gough  (1992).  Giving  feedback  (note  that  the 
workload  assessed  was  the  workload  of  the  person  giving  the  feedback,  not  receiving  it)  increased 
perceived  workload,  instead  of  lowering  it  as  predicted  (although  this  difference  was  not 
statistically  significant).  There  was  also  no  clear  relationship  between  feedback  and  error  rate;  in 
fact,  in  some  conditions  there  was  an  increase  in  errors  with  feedback.  Feedback  appeared  to 
increase  confidence  so  that  people  were  willing  to  make  faster  decisions  on  less  evidence,  meaning 
that  error  rates  increased  even  as  confidence  in  the  decisions  increased.  In  other  words,  feedback 
appears  to  add  to  confirmation  bias.  Feedback  can  increase  perceived  workload,  especially  for  the 
people  who  have  to  provide  it,  and  feedback  can  increase  biases  such  as  confirmation  bias.  As 
workload  is  often  high  in  military  environments,  increasing  workload  will  have  to  be  balanced  with 
any  benefits  provided  by  getting  feedback. 
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5.  Training  implications 


The  previous  sections  have  detailed  the  large  body  of  established  psychological  research,  in 
particular,  related  to  difficulties  and  challenges  in  cue  integration  that  have  been  documented  in  the 
“traditional”  decision  making  literature.  DDM  further  complicates  these  difficulties  through  its 
uncertain,  complex,  changing  nature.  This  points  to  major  challenges  for  military  decision  makers 
performing  DDM;  nonetheless,  proposals  for  improvements  via  training  are  discussed  below. 

What  separates  a  good  decision  maker  from  a  poor  decision  maker  should  be  understood.  This  will 
be  discussed  in  the  context  of  expert  versus  novice  decision  makers.  Once  characteristics  of  good 
versus  poor  decision  makers  are  understood,  it  will  then  be  possible  to  uncover  the  skills  that  may 
be  acquired  to  improve  decision  making  performance.  How  best  to  train  these  skills  in  a  military 
DDM  context  must  also  be  considered. 

5.1  Expert/Novice  Differences 

In  determining  what  to  train  and  how  to  evaluate  training  in  military  DDM,  it  is  useful  to  examine  the 
differences  between  inexperienced  decision  makers  (i.e.,  novices)  and  experienced,  successful 
decision  makers  (i.e.,  experts).  There  appear  to  be  differences  between  novices  and  experts  with 
regards  to  cue  selection  and  judgment  accuracy.  Specifically,  experts  tend  to  be  better  at  cue 
selection,  focusing  on  critical  cues  which  are  more  useful  for  decision  making.  For  example,  Salas  et 
al.  (2001)  argued  that  experts  learn  which  cues  are  critical  cues  for  the  decision  being  made.  One 
similarity  between  experts  and  novices  should  also  be  highlighted.  That  is,  experts  and  novices  appear 
to  select  the  same  number  of  cues  when  making  decisions  (Salas  et  al.,  2001).  As  discussed 
previously  (Section  4.5.1),  one  of  the  challenges  typical  of  DDM  environments  is  that  there  are  a 
large  number  of  cues  which  are  potentially  relevant  for  decision  making.  Experts  do  not  appear  to  use 
more  cues  than  novices,  but  they  are  better  than  novices  at  choosing  which  cues  to  use.  Whether  the 
choice  of  better  cues  is  a  deliberate  one  or  part  of  automatic  processing  is  not  currently  known. 

Research  suggests  that  expertise  also  influences  judgment  accuracy,  although  in  some  cases  the 
relationship  is  negative.  For  example,  in  a  meta-analysis  of  decision-making  studies,  Karelaia  and 
Hogarth  (2008)  found  that  expertise  was  negatively  related  to  judgment  accuracy.  They  noted  that 
different  studies  reported  veiy  different  findings;  in  some  studies  experts  performed  very  well,  and 
in  some  studies  experts  had  veiy  poor  performance.  One  possible  reason  for  this  finding  is  that 
there  was  a  much  higher  error  (noise)  rate  in  studies  which  included  both  experts  and  novices  than 
studies  which  included  only  novices.  The  finding  of  a  negative  relationship  between  expertise  and 
performance  was  made  more  complex  by  the  fact  that  learning  usually  does  occur  over  multiple 
trials  in  the  same  experiment  (i.e.,  in  the  short  term,  experience  improved  performance).  Experts 
also  tend  to  have  more  consistent  judgments  than  novices  (e.g.,  Karelaia  &  Hogarth,  2008). 

Expert  decision  making  also  tends  to  be  more  automatic.  For  example,  Salas  et  al.  (2001)  argued 
that  cues  can  automatically  trigger  appropriate  (or  inappropriate)  behaviours  when  enough  learning 
takes  place.  One  problem  with  such  automatic  decision  making  is  that  experts  cannot  reliably 
report  how  and  why  they  are  making  decisions.  For  example,  Evans  et  al.  (2003)  show  that 
experienced  decision  makers  use  both  explicit  and  implicit  cognitive  processes,  and  so  people  have 
only  partial  conscious  access  to  their  decision  making. 
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There  are  different  types  of  experience  that  a  decision  maker  could  possess;  several  of  these  have 
been  shown  to  make  a  difference  in  decision  making  performance,  particularly  related  to  DDM.  In 
DDM,  experience  is  needed  that  is  related  to  the  task,  the  cues  themselves,  as  well  as  information 
about  the  level  of  expertise  of  the  people  providing  cues  (e.g.,  Brehmer  &  Hagafors,  1986). 

Gonzales  et  al.  (2003)  noted  that  important  aspects  of  learning  which  experts  possess  include 
learning  the  knowledge  base  (declarative  knowledge),  recognition  (ability  to  identify  familiar 
versus  unfamiliar  stimuli),  strategies  (adaptive  production  systems),  and  evaluation  functions 
(ability  to  assess  alternatives).  How  the  decision  maker  determines  when  to  intervene  is  also  highly 
dependent  on  learning. 

Team  situation  awareness  is  often  relevant  in  DDM.  According  to  Salas  et  al.  (2001),  a  critical 
component  of  team  situation  awareness  is  having  knowledge  of  the  informational  requirements  of 
team  members  and  knowing  when  to  exchange  that  needed  information.  Note  that  this  would  likely 
influence  cue  integration  as  this  information  passed  between  team  members  would  be  vital  cues 
used  in  the  decision  making  of  the  team.  Salas  et  al.  also  stressed  the  role  of  experience  (including 
knowledge  of  effective  strategies  in  the  context)  which  is  important  for  interpreting  cues.  The 
authors  stressed  that  it  is  important  both  to  know  what  information  is  critical  and  what  information 
has  to  be  communicated. 

Juslin,  Olsson  et  al.  (2003)  proposed  that  learning  may  take  the  form  of  learning  appropriate 
weights  for  cues  or  from  rule-based  to  exemplar-based  decision  making,  depending  on  the  task 
characteristics  (e.g.,  feedback  type;  repetition  of  exemplars).  Cohen  et  al.  (1999)  discussed  pattern 
recognition  as  one  type  of  expertise;  they  also  argued  that  the  ability  to  create  solutions  when  faced 
with  new  problems  (i.e.,  when  pattern  recognition  would  be  inadequate)  is  also  another 
characteristic  of  expertise. 

In  summary,  expert  decision  makers  tend  to  select  more  appropriate  cues,  have  more  automatic 
decision  making  processes,  and  have  more  consistent  judgment  than  novices.  Thus,  these  skills 
should  probably  be  incorporated  into  military  DDM  training.  However,  there  is  mixed  evidence 
about  judgment  accuracy  between  experts  and  novices,  and  experts  and  novices  do  not  appear  to 
differ  in  the  number  of  cues  used  in  making  decisions.  There  are  many  forms  that  expert 
knowledge  may  take  including  knowledge  of  effective  cue-integration  strategies,  contextual 
information,  and  information  about  cue-criterion  relationships. 

An  important  issue  to  be  considered  when  developing  a  military  DDM  training  program  is  the  fact 
that  experts  may  not  have  conscious  access  to  their  decision-making  processes.  This  lack  of 
conscious  awareness  of  decision-making  processes  indicates  that  understanding  how  experts  differ 
from  novices  cannot  be  achieved  through  means  that  access  conscious  processes  alone  (e.g., 
interviews),  but  that  actual  observations  and  other  means  which  can  reveal  unconscious  factors  and 
processes  will  also  be  required.  Unfortunately,  most  of  the  research  we  reviewed  included  tasks 
that  were  not  performed  in  a  DDM  context;  however,  it  is  likely  that  many  of  the  same  findings 
would  apply  (e.g.,  that  experts  are  generally  better  decision  makers,  show  more  automatic  decision 
making)  but  the  impact  and  relevance  of  these  findings  need  to  explicitly  be  explored  in  the 
military  DDM  context. 

5.2  What  to  train 

There  have  been  many  different  approaches  taken  to  train  cue  integration  processes  in  an  attempt  to 
improve  decision-making  performance.  Training  of  several  aspects  of  cue  integration  has  been 
investigated,  including  training  on: 


Page  42 


Cue  integration  in  Dynamic  Decision  making 


Uumansystems 


HUMAHSYSTEMS 


•  Which  cues  are  important 

•  Thresholds  for  cues  with  continuous  values 

•  Cue  validity 

•  Cue  contingencies  (pattern  recognition) 

•  Mental  models  (background  knowledge) 

•  Critical  thinking 

•  Cue  integration  strategies 

The  importance  of  training  these  different  aspects  of  cue  integration  on  performance  need  to  be 
considered  when  developing  a  military  DDM  training  program. 

Clearly,  learning  which  cues  are  most  relevant  for  decisions  is  likely  to  improve  decision-making 
performance.  For  example,  Salas  et  al.  (2001)  proposed  that  cue-recognition  training  is  a  potential 
training  strategy  for  enhancing  team  situation  awareness.  It  is  intended  to  focus  team  members  on 
the  cues  that  are  most  relevant  for  their  respective  tasks.  This  may  be  a  very  important  strategy 
relevant  for  military  DDM  training,  as  much  of  military  DDM  is  done  in  a  team  context.  Kirlik  et 
al.  (1996)  investigated  the  utility  of  providing  displays  which  emphasized  perceptual  cues,  and 
provided  evidence  that  searching  and  selecting  the  information  within  the  environment  is  often 
difficult.  Methods  that  assist  in  identifying  relevant  cues  within  the  environment  can  significantly 
improve  decision  making  performance.  This  is  particularly  relevant  in  conditions  with  an  increased 
workload.  See  Section  4.4.1  for  a  discussion  of  different  cue  types.  Perceptual  search  is  often  a 
critical  component  of  cue  integration  in  military  DDM  contexts,  both  for  military  personnel  in  the 
field  (e.g.,  searching  for  indications  that  an  enemy  is  nearby)  and  for  operators  (e.g.,  searching  for  a 
new  contact  on  a  radar  display). 

An  additional  factor  on  which  training  could  improve  performance  would  be  training  on  thresholds 
for  cues  with  continuous  values  (rather  than  cues  with  two  discrete  values)  as  this  has  shown  to 
have  improved  performance.  For  example,  Wiggins  and  O’Hare  (2003)  found  an  improvement  in 
determining  when  flight  paths  should  be  changed  based  on  training  in  nine  cues  plus  cue  thresholds 
(e.g.,  sky  darkening  and  degree  of  darkening).  See  Section  4.4.4  for  a  discussion  about  the 
perceptual  challenges  involved  in  discriminating  between  cues  with  continuous  values. 

Although  learning  that  a  cue  is  related  to  a  criterion  is  important,  it  is  unknown  which  aspects  of 
cue-criterion  relationships  should  be  trained  to  produce  the  largest  increases  in  performance. 
Learning  not  only  that  cues  and  criteria  are  related,  but  the  validity  of  that  relationship,  appears  to 
improve  performance.  For  example,  Bois  (2002)  found  that  if  you  provide  participants  with  a 
decision  rule  that  focuses  their  attention  on  proper  cues  and  how  to  weigh  their  importance,  subject 
performance  was  improved,  at  least  for  motivated  participants. 

Whether  relationships  between  single  or  multiple  cues  and  outcomes  should  be  trained  was  a 
question  investigated  by  Biyant  (2005),  in  a  military  context  (contact  classification).  Biyant 
compared  training  of  two  different  types:  pattern-based  training  (typical  patterns  of  cues  are 
learned)  and  cue-based  training  (diagnostic  information  about  each  cue  is  learned).  Both  types  of 
learning  improved  decision-making  performance,  although  slightly  better  performance  was 
obtained  with  cue-based  training. 

Training  in  metacognition  about  cue  integration  and  other  forms  of  critical  thinking  are  likely  to  be 
of  benefit  in  military  DDM.  Background  knowledge  can  also  aid  cue  integration,  including  helping 
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with  cue  selection  and  interpretation.  Thus,  training  in  background  knowledge  such  as  causal 
relationships  and  mental  models  (which  can  contain  causal  relationships)  could  improve  DDM 
performance.  Cohen  et  al.  (1999)  argued  that  optimal  performance  in  DDM  contexts  relies  upon 
appropriate  mental  models  about  organizational  purpose,  own  and  others’  intent,  degrees  of 
initiative,  team  member  reliability,  and  action  sequences.  To  make  the  best  use  of  this  information, 
the  ability  to  think  critically  about  these  mental  models  and  apply  them  to  novel  situations  is  also 
required.  The  focus  of  training  should  therefore  be  the  development  of  a  moderately-sized  set  of 
mental  model  types  (e.g.,  purpose,  intent,  team  member  reliability)  and  critical  thinking  strategies 
that  should  be  used  to  critique  and  correct  those  mental  models  when  pattern  recognition  is 
inadequate  (i.e.,  novel  situations). 

There  are  other  researchers  in  addition  to  Cohen  et  al.  (1999)  who  advocate  the  training  of  critical 
thinking  strategies  related  to  DDM.  Kahneman  (2003)  shows  that  the  interpretation  and  integration  of 
cues  can  be  strongly  influenced  by  the  context,  including  whether  information  is  presented  in  terms  of 
a  gain  or  a  loss  (i.e.,  framing  effects,  see  Sections  4.4.3 .2  and  4.4.3.3  for  further  discussion).  Schultz 
(1 996)  argues  that  training  cannot  change  the  tendency  to  be  influenced  by  perception  of  risk,  but 
critical  thinking  training  might  mitigate  the  effects  on  the  actual  decisions  made. 

There  are  many  different  cue  integration  strategies  which  can  be  used  (see  Section  4.5.2  for  a 
review).  Researchers  have  found  that  the  cue  integration  strategies  used  by  decision  makers  can  be 
influenced  by  training.  For  example  Bryant  (2007)  found  that  participants  could  use  a  resource¬ 
intensive  cue  integration  strategy  when  explicitly  instructed  about  that  strategy.  However,  there  is 
evidence  that  more  resource-intensive  strategies  are  not  necessarily  more  robust  (e.g.,  Dunwoody  et 
al.,  2000).  In  addition,  there  is  significant  evidence  that  expert  decision  makers  do  not  use  more 
resource-intensive  strategies  than  novices.  For  example,  as  already  discussed,  experts  do  not  appear 
to  use  more  cues  than  novices  (e.g.,  Salas  et  al.,  2001).  Thus,  although  it  is  possible  that  effective 
cue  integration  strategies  could  be  trained,  a  thorough  evaluation  of  what  cue  integration  strategies 
are  actually  most  appropriate  for  a  specific  context  will  have  to  be  performed. 

There  is  reason  to  believe  that  training  should  not  focus  on  analytical  strategies  to  the  exclusion  of 
intuitive  strategies.  Dunwoody  et  al.  (2000)  argued  that  intuitive  decision  making  is  more  robust 
than  analytical  decision  making,  as  long  as  errors  can  occur.  Because  of  the  nature  of  errors  made 
in  analytical  versus  intuitive  decision  making,  errors  may  occur  more  frequently  in  intuitive 
decision  making,  but  errors  tend  to  be  closer  to  the  optimal  answer  than  when  errors  occur  in 
analytical  decision  making.  This  is  an  encouraging  result,  as  studies  have  shown  that  military 
decision  making  tends  to  be  more  intuitive  in  nature  (Klein  &  Crandall,  1996). 

In  summary,  many  aspects  of  cue  integration  could  be  taught  in  order  to  improve  military  DDM, 
including  cue-criterion  relationships  and  their  validity,  background  knowledge,  and  cue  integration 
strategies.  Although  some  of  these  different  training  strategies  have  been  compared,  most  have  not 
and  further  research  should  be  done  to  determine  which  aspects  are  most  important  to  decision¬ 
making  performance,  particularly  in  a  military  DDM  context.  For  example,  it  may  enhance 
decision  making  performance  to  perceptually  highlight  important  cues;  however,  this  may  not  be 
feasible  in  uncertain  DDM  tasks  where  there  is  no  overseer  who  knows  what  information  is 
relevant  to  the  decisions  that  need  to  be  made.  Decision  makers  need  to  be  trained  in  order  to 
reduce  their  uncertainty  about  what  is  important  in  the  military  DDM  environment,  and  how 
information  should  be  interpreted.  Thus,  understanding  what  to  train  related  to  cue  integration  in 
military  DDM  requires  more  knowledge  about  the  military  DDM  contexts  themselves  as  well  as 
general  DDM  processes  and  training  methods. 
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5.3  How  to  train 

When  it  is  determined  what  aspects  of  cue  integration  should  be  trained  in  military  DDM,  aspects 
of  the  training  program  itself  must  be  decided.  Issues  include: 

•  Presenting  information 

•  Level  of  difficulty 

•  Providing  feedback 

•  Evaluating  performance 

These  issues  will  be  discussed  in  the  following  sections. 

5.3.1  Presenting  Information 

There  is  evidence  which  suggests  that  information  should  be  actively  processed  and  interacted  with 
during  training.  Klayman  (1988)  found  that  participants  who  were  allowed  to  experiment  in  order 
to  determine  relevant  cues  were  more  successful  decision  makers  than  participants  that  only 
observed.  The  participants  demonstrated  the  ability  to  structure  experiments  during  information 
acquisition  in  a  way  that  made  learning  more  effective.  This  indicates  that  military  DDM  training 
will  likely  be  more  effective  if  trainees  can  learn  in  an  interactive  environment  and  direct  some  of 
their  own  learning  such  as  hypothesis  testing  about  which  cues  are  important  for  a  decision. 

Kirlik  et  al.  (1996)  provided  evidence  that  information  should  be  presented  in  the  form  of  rules, 
and  also  that  decision  makers  can  be  trained  in  making  perceptual  identification  of  important  cues. 
Kirlik  et  al.  investigated  the  benefits  of  rule-based  training  and  perceptual  enhancement. 
Participants  were  asked  to  make  decisions  about  when  the  receiver  in  a  football  game  became 
open.  Groups  were  provided  different  types  of  instruction,  and  further  divided  into  those  who  were 
provided  with  perceptual  enhancement  displays,  and  those  who  were  not.  Results  showed  that 
participant  groups  who  learned  a  list  of  rules  before  training  versus  those  that  did  not  receive  the 
rules  made  more  accurate  decisions.  Furthermore,  those  who  received  the  rules  before  training  and 
used  enhanced  perceptual  displays  performed  the  best.  When  the  perceptual  augmentation 
(highlighting  the  information)  was  removed  after  training,  performance  was  not  diminished; 
however,  how  transferable  this  may  be  to  dissimilar  situations  is  unknown. 

Although  these  findings  were  not  investigated  within  a  DDM  context,  the  findings  are  likely 
relevant  to  military  DDM.  When  training  in  military  DDM,  it  is  reasonable  to  suggest  that  training 
would  be  more  effective  if  decision  makers  interact  with  the  environment  during  training  and  do 
not  simply  passively  observe.  While  it  is  interesting  that  the  use  of  enhanced  perceptual  displays 
improved  decision  making  even  after  the  perceptual  enhancements  were  removed,  the  question  of 
whether  the  important  perceptual  components  of  a  military  DDM  environment  could  be  discovered 
and  enhanced  during  training  has  not  been  adequately  answered.  Likewise,  it  is  not  known  whether 
this  would  be  a  useful  military  DDM  training  strategy.  Likely  the  considerations  of  the  perceptual 
components  of  DDM  would  have  to  be  done  on  a  situational  basis,  as  the  perceptual  components  of 
different  military  DDM  environments  would  likely  be  differentially  relevant  for  decision  making. 

5.3.2  Level  of  Difficulty 

There  appear  to  be  different  levels  of  difficulty  that  should  be  considered  during  the  development 
of  a  military  DDM  training  program.  The  first  is  the  difficulty  of  the  task  itself  (e.g.,  how  many, 
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and  which  cues  are  relevant?),  and  the  second  is  the  complexity  of  the  environment  in  which  the 
task  is  presented  (e.g.,  how  many  irrelevant  cues  are  presented?).  The  difficulty  of  perceptual  tasks 
is  one  area  which  is  likely  to  need  further  study  (see  Section  4.4  for  a  discussion  of  perceptual 
challenges  in  cue  integration). 

There  is  evidence  that  task  difficulty  should  be  increased  during  training.  Salas  et  al.  (2001)  note 
that  the  difficulty  of  the  task  is  important  for  learning;  in  fact,  the  more  difficult  discriminations 
are,  the  more  transfer  actually  takes  place.  However,  there  is  evidence  that  tasks  should  not  begin  at 
an  extremely  difficult  level,  but  difficulty  should  be  increased  during  training.  Costello  (1992) 
reports  the  result  of  an  experiment  using  an  actual  DDM  context  which  allows  interaction  and 
concrete  evaluation  of  performance  (managing  software  development  projects).  It  shows  an  effect 
of  the  anchoring  and  adjustment  heuristic  on  cue  integration,  situations  under  which  it  may  have  a 
greater  effect,  and  some  factors  which  may  influence  overall  performance.  Performance  appeared 
to  be  better  when  the  easier  of  the  two  projects  was  managed  first.  This  result  supports  a  training 
method  in  which  difficulty  is  increased. 

Kass,  Herschler,  and  Companion  (1991)  provided  evidence  that  simplifying  the  environment 
during  training  can  improve  performance,  even  when  tasks  are  performed  in  complex  environments 
during  testing.  Participants  were  trained  to  identify  tank  muzzle  flash  patterns  in  both  simplified 
(i.e.  only  the  opposing  tank  in  the  horizon  and  no  sound)  and  complex  (i.e.  sound  was  on  and  other 
vehicles  were  present)  environments.  When  later  tested  in  the  complex  environment,  subjects 
trained  in  the  simplified  environment  were  the  top  performers.  Kass  et  al.  (1991)  suggested  that  the 
better  performance  was  due  to  increased  stimulus-response  matching  efficiency,  causing  increased 
cue  salience  in  the  complex  test  condition.  These  results  counter  state-dependent  learning  theories 
which  suggest  that  performance  is  better  when  training  and  testing  environments  are  similar. 

One  of  the  challenges  of  military  DDM  environments  is  that  they  are  often  complex;  therefore 
understanding  how  complexity  should  be  controlled  during  training  is  highly  relevant.  Evidence 
suggests  that  introducing  complexity  during  training  is  important  so  that  training  can  be  transferred 
to  complex  environments;  however,  training  on  a  simplified  environment  also  shows  promise. 
Applying  this  research  to  military  DDM  training  would  require  further  research  to  determine  which 
aspects  of  the  military  environment  are  critical  for  DDM  and  which  are  simply  extraneous  and 
adding  to  the  difficulty  of  the  task.  Without  such  knowledge,  it  would  be  difficult  to  create  a  DDM 
training  program  that  could  effectively  use  a  simplified  military  environment. 

5.3.3  Providing  Feedback 

How  to  present  feedback  during  military  DDM  training  is  a  key  issue.  Feedback  is  important  for 
learning,  as  decision  makers  must  understand  how  their  actions  affect  the  environment.  However, 
as  previously  discussed  (see  Section  1.4.1 .7)  there  is  often  either  inadequate  feedback  presented, 
and  decision  makers  often  have  difficulty  using  the  feedback  that  they  do  get. 

Feedback  can  be  provided  either  continuously  (i.e.,  feedback  for  every  action  can  be  provided)  or 
occasionally.  Klayman  (1988)  provides  evidence  that  occasional  feedback  can  lead  to  effective 
learning,  but  there  is  no  direct  comparison  with  continuous  feedback. 

Recall  from  Section  4.7  that  the  type  of  feedback  presented  is  also  critical  for  improving 
performance.  Information  about  relations  between  cues  and  criteria  seems  to  be  the  most  important 
type  of  feedback  for  improving  performance  (Balzer  et  al.,  1992).  Information  about  the 
participant’s  cognitive  system  or  actual  performance  results  does  not  appear  to  be  as  helpful. 
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As  well  as  the  content  of  the  feedback,  the  method  in  which  feedback  is  presented  also  appears  to 
influence  how  well  it  can  be  used  to  improve  performance.  Providing  information  verbally  about 
relevant  cues  does  not  appear  to  be  the  best  method  for  providing  feedback.  For  example,  Salas  et  al. 
(2001)  found  that  simply  verbally  coaching  people  about  which  cues  are  relevant  does  not  improve 
their  cue  selection;  however,  highlighting  physical  cues  on  a  visual  display  can  improve  performance. 

The  proper  interpretation  of  feedback  is  usually  a  difficult  process  in  military  DDM.  This  difficulty 
is  at  least  partially  due  to  the  fact  that  there  are  delays  between  the  actions  of  the  decision  maker  and 
the  impacts  on  the  environment,  and  also  due  to  the  fact  that  effects  can  be  probabilistic  (i.e.,  the 
same  actions  don’t  always  have  the  same  effects).  Although  the  research  reviewed  here  provides 
some  evidence  that  feedback  does  not  have  to  be  continuous  to  be  effective,  and  that  feedback  about 
the  relationships  between  cues  and  criteria  should  be  provided,  much  more  research  needs  to  be  done 
to  investigate  the  best  way  to  present  feedback  to  facilitate  its  use  for  military  DDM  training. 

5.3.4  Evaluating  performance 

How  to  evaluate  performance  during  military  DDM  training  is  an  important  question,  and  may  not 
be  as  straightforward  as  one  might  assume.  For  example,  should  performance  be  evaluated  in  a 
certain  situation  or  a  wide  variety  of  situations  (i.e.,  robustness)?  How  can  actual  performance  be 
evaluated  (i.e.,  what  should  be  used  as  a  benchmark)?  An  extremely  important  issue  during  military 
DDM  training  program  development  is  balancing  the  desired  expertise  of  the  trainees  with  the 
amount  of  resources  which  can  be  devoted  to  their  training.  These  issues  are  discussed  below. 

There  are  several  researchers  who  have  argued  that  robustness  should  be  the  goal  of  training. 
Busemeyer  and  Pleskak  (2008)  argued  that  DDM  decision  makers  should  be  trained  in  a  way  that 
increases  robustness  (i.e.  in  strategies  that  are  suitable  across  a  wide  variety  of  options  due  to 
variability  in  the  way  the  system  could  evolve)  rather  than  for  optimality  given  the  current  or 
predicted  situation.  It  is  also  possible  that  robustness  across  tasks  should  be  considered,  as  different 
output  requirements  may  influence  cue  integration.  For  example,  requiring  a  categorical  (e.g.,  a 
yes/no  response;  is  there  an  enemy  behind  the  next  hill?)  response  versus  a  probabilistic  response 
(e.g.,  what  is  the  likelihood  that  there  will  be  an  enemy  behind  the  next  hill?)  can  affect  the  number 
of  cues  used  to  make  a  decision  and  how  they  are  integrated  (e.g.,  Juslin,  Olsson  et  al.,  2003;  for 
further  discussion  of  task  effects  in  cue  integration,  see  Section  4.4.2).  There  are  situations  in  which 
categorical  or  probabilistic  responses  might  be  the  most  useful  in  a  military  DDM  context;  thus, 
which  type  of  decision  is  required  should  be  taken  into  account  when  creating  a  training  program. 

Dun  woody  et  al.  (2000)  argued  that  analytical  decision  making  is  often  used  as  a  benchmark  or 
“ceiling”  for  performance,  and  that  this  is  not  actually  a  good  approach.  It  is  often  assumed  that 
analytical  processes  can  be  used  as  a  “ceiling”  for  performance,  and  participants’  performance  can 
be  evaluated  with  respect  to  that,  presumably  optimal,  performance.  However,  Dunwoody  et  al. 
argued  that,  because  of  the  nature  of  errors  made  in  analytical  versus  intuitive  decision  making, 
errors  may  occur  more  frequently  in  intuitive  decision  making,  but  errors  tend  to  be  closer  to  the 
optimal  answer  than  when  errors  occur  in  analytical  decision  making;  thus,  intuitive  decision 
making  tends  to  be  more  robust.  Therefore,  assuming  that  analytical  decision  making  should  be 
considered  as  a  “ceiling”  for  performance  is  not  correct  if  errors  in  decision  making  can  occur.  This 
is  consistent  with  the  findings  of  Klein  (1998;  cited  in  Kahneman,  2003)  who  found  that  under 
pressure,  experienced  decision  makers  do  not  necessarily  need  to  choose  between  options  as 
generally  only  one  option  comes  to  mind.  Thus,  it  may  be  important  in  military  DDM  training  to 
emphasize  that  intuitive  decision  making  is  not  to  be  avoided. 
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In  considering  the  amount  of  resources  to  devote  to  training,  Bakken  (2008)  notes  that  different  levels 
of  training  require  different  amounts  of  time  and  other  resources,  so  an  assessment  of  the  required 
level  of  task  competency  need  to  be  made  before  a  training  program  is  designed.  He  discusses  “high 
road”  versus  “low  road”  training,  and  uses  the  medical  profession  as  an  example.  A  “high  road” 
approach  is  geared  toward  producing  an  expert  surgeon  which  requires  about  10  years  of  schooling 
and  an  additional  1 0  years  of  performing  under  supervision  before  true  mastery  is  obtained.  In 
contrast,  a  “low  road”  approach  is  similar  to  someone  learning  first  aid  -  very  basic  training  is 
supplied,  the  time  and  resource  commitment  is  relatively  low,  but  the  person  is  only  limited  in 
competence.  As  the  amount  of  desired  skill  increases  and  the  amount  of  possible  time  to  devote  to 
learning  increases,  the  need  to  spend  time  initially  on  analytical-type  processing  (which  will  gradually 
become  intuitive  with  practice)  increases.  For  example,  an  inability  to  incorporate  feedback  has  to  be 
overcome,  but  the  extent  to  which  this  is  required  depends  on  the  desired  level  of  skill. 

Note  that  there  are  many  challenges  in  military  DDM  which  should  be  considered  when  evaluating 
performance.  For  example,  in  many  of  the  experiments  reviewed  here,  participants  were  given  as 
much  time  as  they  desired  to  make  relatively  simple  decisions.  In  military  DDM  environments,  there 
are  often  time  factors  which  are  challenging;  decisions  may  have  to  be  made  quickly,  and  there  may 
be  vigilance  components  such  that  decisions  must  be  implemented  at  exactly  the  right  time. 

5.4  Training  Cue  Integration  in  Military  DDM 

Applying  the  research  related  to  training  cue  integration  in  decision  making  to  a  military  DDM 
context  is  not  a  simple  undertaking.  In  many  cases,  factors  investigated  in  the  literature  are  not 
likely  to  be  available  in  a  military  DDM  context  (e.g.,  using  actual  cue  validities  to  train 
participants).  However,  there  are  several  important  factors  that  have  been  highlighted  and  these 
should  be  considered. 

Perhaps  the  most  important  is  the  consideration  of  how  many  resources  are  available  for  training 
and  how  proficient  the  decision  maker  is  expected  to  be  once  training  is  complete.  These  questions 
will  likely  have  to  be  answered  separately  for  different  military  DDM  contexts. 

Evaluation  of  performance  is  another  main  issue  which  will  need  to  be  considered.  Is  knowledge  of 
“ground  truth”  possible?  If  not,  then  providing  people  with  information  about  facts  such  as  cue 
validity  would  not  be  possible.  Furthermore,  should  people  be  trained  to  make  robust  predictions, 
or  is  analytical  reasoning  preferred? 

Understanding  what  needs  to  be  trained  to  improve  military  DDM  will  also  have  to  be  addressed;  this 
again  will  be  affected  by  the  specific  task  and  context  being  trained.  One  important  issue  involves 
understanding  what  makes  a  decision  maker  effective  in  a  particular  task.  For  example,  in  a  team- 
based  situation,  understanding  the  roles  of  teammates  is  likely  to  positively  impact  performance,  but 
this  would  be  irrelevant  if  decision  makers  act  independently.  Because  much  of  military  DDM  occurs 
in  a  team  context,  team  knowledge  is  likely  to  play  an  important  role,  and  therefore  explicit  training 
about  teammates’  roles  may  prove  valuable  and  significantly  improve  military  DDM. 

As  previously  mentioned,  it  is  difficult,  and  may  be  impossible,  to  know  ahead  of  time  which  cues 
will  be  important  for  making  particular  decisions  within  military  DDM.  This  is  partially  the  result 
of  military  DDM  involving  multiple  groups  of  intelligent,  autonomous  agents.  One  of  the 
implications  of  the  fact  that  military  DDM  usually  involves  an  opposing  force  or  forces  is  that  the 
enemy  generally  wishes  to  remain  as  unpredictable  as  possible,  becoming  adaptive  adversaries.  For 
example,  insurgents  in  Afghanistan  who  plant  Improvised  Explosive  Devices  (IEDs)  always  want 
to  plant  them  in  new  ways  and  therefore  avoid  IED  detection  until  the  weapons  can  be  used 
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effectively.  Thus,  once  the  military’s  ability  to  forecast  becomes  somewhat  accurate,  the  enemy  is 
motivated  to  change  their  behaviour  such  that  the  old  cues  that  were  predictive  will  no  longer  be 
useful  (or  even  result  in  actions  which  previously  would  have  been  effective  resulting  in  poor 
outcomes).  Because  of  the  difficulty  of  predicting  which  cues  will  be  useful  in  decision  making,  it 
is  likely  that  military  DDM  training  should  incorporate  critical  thinking  (metacognitive)  skills,  as 
well  as  low-level  training  in  cue  perception  and  selection.  Training  about  which  cues  are  likely  to 
be  useful  will  probably  benefit  the  decision  maker,  but  in  a  military  DDM  context  such  skills  need 
to  be  paired  with  critical  thinking  skills  to  encourage  decision  makers  to  take  a  broader  view  and 
not  blindly  use  the  cues  and  cue  integration  strategies  which  have  previously  worked. 
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6.  Conclusions 


There  is  a  great  deal  of  potential  benefit  to  designing  a  training  program  for  CF  military  decision 
makers  that  explicitly  seeks  to  improve  cue  integration.  Before  summarizing  some  specific  training 
suggestions,  cognitive  challenges  associated  with  the  steps  of  the  information  processing  model 
(Wickens  &  Carswell,  2006)  are  reviewed  below  with  explicit  linkages  back  to  military  DDM 
contexts. 

Perception 

Dynamic  decision  makers  are  extremely  vulnerable  to  the  way  in  which  information  is  presented. 
This  means  that  important  cues  that  are  not  accessible/salient  will  most  likely  be  ignored.  Whether 
training  can  be  effectively  applied  to  overcome  this  is  unknown. 

In  general,  different  types  of  cues  affect  decision  making  differently  -  with  evidence  for  sub¬ 
conscious  effects  being  prevalent  but  not  well  understood.  For  our  purposes  internal  cues  (i.e.  cues 
from  within  a  decision  maker)  versus  external  cues  (i.e.  those  in  the  environment  around  a  decision 
maker)  were  discussed.  Notably,  affect  is  an  internal  cue  that  plays  a  major  role  in  how  decision 
makers  perceive  their  world.  Preliminary  studies  indicate  that  this  is  a  key  cue  in  decision  making; 
however,  its  application  to  DDM  has  not  been  fully  explored. 

How  cues  are  selected  remains  elusive,  with  unimportant  cues  affecting  decision  making,  and  the 
information  presentation  of  cues  playing  a  crucial  role  in  cue  selection.  Further  study  of  the  cue 
selection  process  is  warranted,  specifically  in  military  DDM  scenarios.  Specifically,  “gain”  versus 
“loss”  framing  is  an  external  cue  that  seems  to  have  unique  effects  on  decision  making.  Applying 
the  gain/loss  framing  effect  to  military  DDM  applications  should  have  strong  implications  on  all 
levels  of  military  DDM  from  tactical  up  to  strategic  (e.g.,  a  losing  commander  may  be  inclined  to 
accept  more  risks  that  one  who  is  winning,  Schultz,  1996). 

Working  memory 

There  exist  an  extremely  large  number  of  cues  in  dynamic  decision  making  environments.  This 
includes  informational  cues  from  the  environment,  as  well  as  cues  about  cues  (i.e.  cues  about  how 
valid  other  cues  are).  All  this  leads  to  potential  overloads  of  working  memory.  Thus,  it  is  postulated 
that  decision  makers  have  developed  strategies  to  integrate  the  cues  available.  Take-the-best  (and 
variations  on  this  approach)  are  discussed.  Research  suggests  that  these  strategies  are  not  optimal, 
do  not  weight  cues  perfectly,  and  that  changing  cue  values  are  hard  to  incorporate.  Yet,  these 
strategies  appear  to  be  efficient  and  successful  approaches  to  meet  day  to  day  decision  making 
challenges  (Klein  &  Crandall,  1996).  The  extent  to  which  these  strategies  are  applied  in  DDM  has 
not  been  fully  researched;  however,  studies  showing  far  from  optimal  results  in  DDM 
environments  suggest  that  whatever  cue  integration  strategies  are  being  applied  are  not  very 
successful  (Brown  et  al.,  2009). 

Further,  previous  research  on  the  use  of  mental  models  in  DDM  is  discussed.  Mental  models 
appear  to  be  a  way  in  which  decision  makers  might  be  able  to  simplify  their  environment  (Brown  et 
al.,  2009).  Although  there  is  a  vast  literature  on  mental  models,  very  little  literature  exists  that 
provides  a  description  of  the  cue  integration  processes  involved  in  mental  models. 
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Long  term  memory 

The  literature  we  reviewed  related  to  long-term  memory  and  cue  integration  did  not  involve  DDM 
tasks.  However,  it  is  reasonable  to  suggest  that  the  effects  of  forgetting,  biased  recall,  difficulties  in 
deliberate  recall,  and  unintentional  recall  found  would  be  similar  (if  not  in  made  worse  so)  in  DDM 
environments. 

Further,  cues  that  change  over  time  are  a  central  part  of  DDM.  This  means  that  cues  have  to  be 
accessed  and  changed  in  long  term  memory  for  successful  decision  making.  However,  studies  show 
that  once  a  cue-outcome  relationship  has  been  learned,  learning  a  new  relationship  between  a 
different  cue  and  the  same  outcome  interferes  with  the  previous  learning.  More  research  needs  to  be 
done  to  understand  under  what  conditions  cue-criterion  relationship  changes  are  actually  detected, 
and  under  what  conditions  previous  cue-criterion  relationships  are  deemed  to  be  relevant  again. 

Finally,  causal  knowledge  is  used  as  a  context  for  understanding  the  relationships  between  cues  and 
criteria.  The  use  of  causal  knowledge  may  be  partially  controlled,  but  there  is  also  evidence  that  the 
use  of  causal  knowledge  is  not  under  complete  conscious  control.  Therefore  causal  reasoning  may 
play  an  important  role  in  cue  integration  and  may  not  be  amenable  to  changes. 

Feedback 

Feedback  is  critical  for  learning  the  relationship  between  actions  and  their  consequences.  However, 
in  DDM  feedback  delays  as  well  as  ambiguous  feedback  can  complicate  this  learning  process.  In 
addition  to  the  actual  lack  of  unambiguous  feedback  and  feedback  delays  in  DDM,  it  has  also  been 
found  that  decision  makers  do  not  seem  to  make  ideal  use  of  the  feedback  that  is  available. 

One  type  of  feedback  that  has  been  found  to  improve  performance  is  cognitive  feedback.  This 
means  that  if  decision  makers  can  be  provided  instruction  about  the  underlying  relationships 
between  variables  in  DDM  environments  (rather  than  being  forced  to  determine  the  relationships 
themselves)  DDM  performance  should  theoretically  improve. 

6.1  Training 

Finally,  implications  of  these  factors  on  military  training  in  a  CF  context  are  discussed.  Building  on 
the  previous  knowledge,  we  used  recent  empirical  research  to  infer  opportunities  for  training 
military  decision  makers  in  dynamic  decision  making  environments.  Many  aspects  of  cue 
integration  could  be  taught,  including  cue-criterion  relationships  and  their  validity,  background 
knowledge,  and  cue  integration  strategies. 

Evidence  suggests  that  introducing  complexity  during  training  is  important  so  that  training  can  be 
transferred  to  complex  environments;  however,  training  on  a  simplified  environment  also  shows 
promise.  Applying  this  research  to  DDM  training  would  require  further  research  to  determine 
which  aspects  of  the  environment  are  critical  for  DDM  and  which  are  simply  extraneous  and 
adding  to  the  difficulty  of  the  task.  Without  such  knowledge,  it  would  be  difficult  to  create  a 
training  program  that  could  effectively  use  a  simplified  environment. 

There  are  additional  limitations  on  training  in  military  DDM  situations  specifically.  Notably,  the 
need  to  know  “ground  truth”  is  required  in  order  to  train  many  things  (e.g.,  important  cues  and 
when  these  cues  change);  however,  this  ground  truth  knowledge  may  be  impossible  to  come  by  in 
military  DDM.  In  general,  military  DDM  decision  makers  should  be  trained  in  a  way  that  increases 
robustness  (i.e.  in  strategies  that  are  suitable  across  a  wide  variety  of  options  due  to  variability  in 
the  way  the  system  could  evolve)  rather  than  for  optimality  given  the  current  or  predicted  situation 
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(Busemeyer  &  Pleskak,  2008).  This  most  likely  means  that  training  should  entail  a  combination  of 
strategies  for  cue  integration  and  critical  thinking  as  no  one  solution  is  likely  to  adequate  in 
military  DDM.  However,  by  affording  decision  makers  the  proper  strategies  and  knowledge  about 
when  to  apply  them  performance  should  improve. 
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7.  Future  Research  Recommendations 


In  general  there  was  not  a  sizable  amount  of  work  relating  to  cue  integration  in  DDM  found.  Thus, 
any  future  research  in  this  area  is  encouraged.  A  number  of  suggestions  are  made  concerning  more 
specific  future  research  areas  to  advance  the  knowledge  of  how  cue  integration  is  performed  and 
can  be  improved  in  DDM  situations. 

7.1  Future  research  considering  internal  cues 

Looking  at  the  broader  picture  of  cognitive  functioning,  the  role  of  consciousness  in  cue  integration 
may  be  considered.  Within  the  information  processing  model,  consciousness  is  complicated.  At  this 
point,  it  is  unknown  whether  the  selection  of  cues  is  a  deliberate  or  automatic  process,  and  this  is 
possibly  a  topic  for  future  research. 

Along  a  similar  perspective,  the  role  of  attention  within  cue  integration  was  not  found  within  the 
literature  but  could  prove  to  be  a  valid  area  of  research.  Specifically,  the  role  of  attention  in 
selection  of  cue  is  especially  applicable  to  DDM. 

The  role  of  cue  integration  as  it  is  associated  to  mental  models  was  briefly  discussed.  Cue 
integration  seems  to  be  a  significant  component  of  developing  and  maintaining  updated  mental 
models  that  are  used  within  DDM.  Examining  this  relationship  in  further  detail  however,  was 
beyond  the  scope  of  this  project.  It  may  be  that  mental  models  are  a  moderating  mechanism 
through  which  cue  integration  functions.  Future  research  may  want  to  consider  investigating  the 
possible  relationships  between  these  concepts  and  their  role  in  DDM. 

Finally,  the  role  of  military  decision  maker’s  motivations  and  goals  on  cue  integration  should  be 
explored.  It  is  presumed  that  with  increased  motivation,  the  decision  maker  is  more  vigilant  and 
attentive  to  cues;  however,  verification  of  this  (and  its  assumed  positive  effect  on  the  resultant  cue 
integration  process)  is  required.  Building  on  this  within  the  CF,  the  relationship  between 
commander’s  intent,  common  intent  and  cue  integration  should  prove  interesting.  The  role  of  affect 
in  cue  integration  would  also  be  relevant  to  this  study. 

7.2  Future  research  considering  external  cues 

Framing  effects  have  a  distinctive  role  in  establishing  particular  risk  within  a  decision.  Applying  a 
gain/loss  framing  effect  to  military  applications  would  have  strong  implications  on  all  levels  of 
military  decision  making  from  tactical  up  to  strategic.  It  may  be  beneficial  to  further  examine  the 
role  of  framing  effects  within  different  military  operations. 

The  literature  highlights  the  dynamic  aspects  of  DDM.  Within  this,  new  cues  that  are  gathered  must 
be  integrated  into  the  decision  process.  This  can  introduce  and  change  the  relationships  of  cues. 
More  research  needs  to  be  done  to  understand  under  what  conditions  cue-criterion  relationship 
changes  are  actually  detected,  and  under  what  conditions  old  cue-criterion  relationships  are  deemed 
to  be  relevant  again. 

A  second  component  of  cue  relationships  involves  the  constantly  changing  nature  of  cues.  The 
temporal  aspect  of  cues  including  the  introduction  of  new  cues,  changing  of  cues  values,  and  the 
changing  relationships  between  cues,  creates  significant  challenges  in  interpretation  and 
integration.  This  is  especially  an  issue  in  military  decision  making,  when  enemy  units  are 
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continually  on  the  move  and  other  external  events  constantly  change  the  lay  of  the  land  upon  which 
missions  are  being  planned.  Yet  there  was  no  literature  found  discussing  the  best  ways  to  provide 
military  decision  makers  with  skills  to  better  deal  with  the  temporal  nature  of  cues.  Thus,  the 
temporal  nature  of  cues  is  an  important  area  for  future  research. 

The  nature  of  different  DDM  tasks  influences  decision-making  processes.  DDM  contexts  should  be 
examined  to  determine  how  potential  task  types  and  the  implications  of  these  different  tasking 
types  influences  optimal  information  presentation  and  the  types  of  processing  that  need  to  be 
supported.  A  CF  focused  task  typology  could  even  be  created  to  further  develop  and  detail  the 
nature  of  different  DDM  tasks.  This  would  include  examining  the  cue  selection  process  to 
determine  how  and  why  certain  cues  are  used  and  others  are  not. 

In  discussing  training,  a  comparison  of  the  effectiveness  in  training  cue-criterion  relationships  and 
their  validity,  background  knowledge,  and  cue  integration  strategies  in  a  DDM  context  would  be 
beneficial.  As  well,  research  is  required  to  determine  which  aspects  of  the  environment  are  critical 
for  DDM  and  which  are  simply  extraneous  and  adding  to  the  difficulty  of  the  task.  Without  such 
knowledge,  it  would  be  difficult  to  create  a  training  program  that  could  effectively  use  a  simplified 
environment  and  maintain  appropriate  transfer  to  a  dynamic  context. 

Also  for  training,  the  importance  of  feedback  in  learning  remains  a  top  priority.  Thus,  proper 
methods  for  the  evaluation  of  feedback  should  be  researched.  For  DDM,  decision  makers  need  to 
understand  the  impact  that  their  decisions  have  on  the  environment.  Though  this  is  acknowledged 
and  documented,  enabling  decision  makers  to  properly  use  feedback  remains  illusive. 
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Annex  A 


The  article  reviewed  in  the  first  phase  of  this  project  are  presented  below  in  alphabetical  order: 

1 .  Title:  Examining  the  effect  of  information  sequence  on  patriot  air  defense  officers'  judgments 
Author(s):  Adelman,  L.  &  Bresnick,  T.  A. 

Source:  Report  prepared  for  the  United  States  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  ARI  Report  No.  95-23. 

Year:  1995 _ 

Abstract: 

This  paper  describes  a  recent  experiment  conducted  with  Patriot  air  defense  officers  and  using  the 
Patriot  air  defense  simulators  at  Fort  Bliss,  Texas.  The  experimenters  found  that,  under  certain 
conditions,  the  participants  made  different  identification  judgments  and  took  different  engagement 
actions  depending  on  the  sequence  in  which  the  same  information  was  presented  to  them.  This 
finding  was  consistent  with  theoretical  predictions  regarding  how  operators  process  information 
and  the  hypothesis  that  their  processing  approach  (or  Heuristic)  would  result  in  biased  judgments 
under  certain  conditions.  Future  research  is  directed  toward  investigating  whether  display 
modification  can  remove  the  observed  judgmental  bias.  Generally,  this  experiment  demonstrates 
the  applied  implications  of  basic  research  investigating  human  information  processing,  and  the 
importance  of  understanding  cognitive  processes  when  developing  computer  systems. 

Notes: 

Relevance:  discusses  heuristics  (anchoring-and-adjustment  heuristic),  biases,  and  cue  integration 

The  goal  of  the  authors  is  to  show  that  judgments  of  probability  and  chosen  actions  can  change 
depending  on  the  order  in  which  information  is  presented. 

This  article  examines  heuristics  and  bias  in  decision  making,  particularly  those  which  may  be 
related  to  the  use  of  computer  systems  by  military  personnel.  The  particular  heuristic  investigated 
was  a  version  of  the  “anchoring-and-adjustment”  heuristic  related  to  decisions  about  “friend  vs. 
foe”  identification  judgments. 

The  authors  seem  to  have  an  illogical  interpretation  of  the  effects  of  new  information  on  an  anchor. 
They  state  that  “Since  each  new  piece  of  information  leads  to  a  new  anchor,  recent  information  is 
weighted  more  than  prior  information”  (p.  2),  which  is  not  correct  according  to  the  traditional 
anchoring-and-adjustment  heuristic  -  the  anchor  is  the  first  piece  of  information,  and  new 
information  is  the  “adjustment”  portion  of  the  heuristic,  and  acts  to  adjust  the  position.  The  idea  is 
that  the  initial  anchor  has  a  large  impact  on  the  position,  which  is  the  opposite  interpretation  that 
provided  by  Adelman  &  Bresnick,  who  argue  that  new  information  will  have  a  larger  impact  on  the 
position.  Because  Adelman  &  Bresnick  refer  to  a  model  by  Hogarth  and  Einhom  which  is  not 
reviewed  here,  it  is  not  known  whether  Adelman  &  Bresnick  misunderstand  the  model,  or  whether 
Hogarth  and  Einhom  have  a  different  interpretation  than  the  traditional  one  (i.e.,  by  Kahneman  & 
Tversky). 

The  authors  seem  to  also  be  ignoring  the  confirmation  bias  -  they  state  that  the  more  certain 
someone  is  of  a  conclusion,  the  larger  effect  discontinuing  evidence  will  have  on  the  judgment. 
This  is  not  what  is  typically  found,  and  the  authors  do  not  provide  an  adequate  rationale  for  this 
prediction. 
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Another  problem  with  this  analysis  is  that  they  use  difference  scores  (from  one  probability 
judgment  to  another).  This  means  that  floor/ceiling  effects  could  be  at  play  and  there  would  be  no 
way  to  know  this,  since  the  absolute  proportion  judgments  are  not  provided.  Another  issue  is  that 
there  are  multiple  analyses  performed  without  apparent  control  for  experiment-wise  error,  and  only 
a  few  significant  results,  leading  the  reader  to  guess  that  some/all  of  the  significant  findings  might 
be  a  result  of  experiment-wise  error.  Strengthening  this  belief  is  the  fact  that  no  significant  results 
were  found  in  an  ambiguous  condition  which  would  likely  have  resulted  in  the  most  room  for 
changes  in  probability  judgments,  and  therefore  would  be  suspected  to  lead  to  the  larger  order 
effects. 

There  are  serious  flaws  in  this  research;  however,  there  may  be  some  points  worth  considering. 

One  of  the  challenges  in  supporting  DDM  would  be  deciding  how  to  present  information.  One 
choice  that  would  have  to  be  made  is  whether  information  should  be  presented  serially  or  be 
available  all  at  once.  The  article  supports  the  idea  that  performance  may  be  marginally  better  when 
information  is  presented  all  at  once;  there  were  some  effects  of  information  order  which  led 
participants  to  incorrectly  classify  a  friendly  aircraft  as  unfriendly  or  choose  to  shoot  down  a 
friendly  aircraft.  However,  this  leaves  the  issue  of  whether  working  memory  can  cope  with  all  of 
the  information  that  would  be  required  to  be  processed  in  other  DDM  scenarios. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  HEURISTICS,  BIASES,  AND/OR  CUE 
INTEGRATION _ 

To  be  used?  INCLUDE  IF  RELEVANT  -  note  that  the  ideas  here  are  very  similar  to  the  other 
Adelman  paper  -  this  one  is  more  recent 


2.  Title:  Examining  the  effect  of  information  order  on  expert  judgment 
Authors):  Adelman,  L.,  Tolcott,  M.  A.,  &  Bresnick,  T.  A. 

Source:  Report  prepared  for  the  United  States  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  ARI  Report  No.  91-78. 

Year:  1991 _ 

Abstract: 

ABSTRACT  DIFFERENT  ON  WEBPAGE  AND  IN  REPORT  -  REPORT  ABSTRACT 
FOLLOWS: 

For  this  report,  researchers  performed  an  experiment  in  which  trained  Army  air  defense  personnel 
created  a  paper-and-pencil  representation  of  their  task  to  test  the  predictions  of  the  Einhom- 
Hogarth  model  for  belief  updating.  For  a  simple  task  with  a  short  series  of  information,  the  model 
predicted  a  significant  information  order  by  response  mode  interaction.  The  experiment  supported 
the  model’s  predictions.  When  information  was  presented  sequentially  and  a  probability  estimate 
was  obtained  after  each  piece  of  information,  the  order  in  which  the  same  information  was 
presented  significantly  affected  the  final  mean  probability  estimates.  In  contrast,  when  all  the 
information  was  presented  at  once  and  a  probability  estimate  was  obtained  at  that  time,  information 
order  had  no  effect.  There  were,  however,  significant  individual  differences.  Moreover,  order  and 
response  mode  cumulatively  accounted  for  less  than  10  percent  of  the  total  variation  in  the 
participants’  probability  estimates.  These  findings,  and  their  implications,  are  discussed  in  the 
report. 
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Notes: 

The  authors  examine  the  effect  of  sequential  information  presentation  versus  concurrent 
information  presentation.  They  predict  effects  based  on  a  model  adapted  from  the  anchoring-and- 
adjustment  heuristic,  which  fundamentally  differs  from  the  “traditional”  anchoring-and-adjustment 
heuristic.  The  authors  are  examining  the  effect  of  different  information  orders  when  the  beginning 
“anchor”  is  the  same,  rather  than  examining  the  effect  of  the  same  information  with  a  different 
beginning  value.  Thus,  it  is  difficult  to  argue  that  the  significant  evidence  backing  the  traditional 
anchoring-and-adjustment  heuristic  is  relevant  to  the  logic  here. 

There  are  other  problems  with  this  study,  including  the  possibility  of  sequential  effects  between 
trials  (all  participants  complete  all  trials  and  do  so  in  the  same  order  within  the  two  conditions 
[sequential  vs.  concurrent  information  presentation]),  and  possible  floor/ceiling  effects  contributing 
to  differences  in  probability  adjustment  with  added  information.  They  also  perform  a  large  number 
of  analyses  without  apparently  accounting  for  experiment-wise  error. 

Although  there  are  fundamental  flaws  in  this  study,  it  does  suggest  that  there  are  differences  in  the 
conclusion  arrived  at  when  information  is  presented  in  different  orders  versus  concurrently.  Due  to 
the  large  amount  of  information  that  generally  has  to  be  integrated  in  a  DDM  scenario, 
understanding  how  information  order  affects  information  aggregation  should  be  considered.  This 
would  be  relevant  both  for  training  and  designing  support  systems  to  facilitate  information 
aggregation  and  decision  making. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  HEURISTICS,  BIASES,  AND/OR  CUE 
INTEGRATION _ 

To  be  used?  FOR  CONSIDERATION  -  Note  that  these  ideas  are  very  similar  to  the  other 
Adelman  paper  and  that  one  is  more  recent 


3.  Title:  On  improving  dynamic  decision-making:  Implications  from  multiple-process  cognitive 
theory 

Author(s):  Bakken,  B.  E. 

Source:  Systems  Research  and  Behavioral  Science,  25,  493-501 

Year:  2008 _ 

Abstract: 

Decision  environments  that  afford  unambiguous  and  transparent  feedback  allow  humans  to  build 
up  corresponding  mental  models  that  give  them  good  decision  guidance.  But  in  many  complex 
decision  situations,  such  feedback  is  not  available.  In  such  environments  people  not  only  show 
initial  misperceptions,  but  also  fail  to  learn.  Education  and  training  have  therefore  been  proposed, 
but  with  limited  success.  This  paper  applies  multiple-process  cognitive  theory  to  explain  findings 
of  poor  learning  and  sub-optimal  decision-making.  The  applied  theory  suggests  that  intuitive 
processes  are  default  in  decision-making  and  natural  learning.  Analytic  processes  are  more  seldom 
applied.  Implications  for  education  and  training  in  dynamic  decision-making  are  suggested,  and  in 
light  of  the  primacy  of  intuitive  processes,  include  massive  exposure  to  complex  dynamic  decision¬ 
making  in  order  to  improve  upon  intuition  in  such  environments. 

Notes: 

The  author  points  out  that  to  learn  how  to  perform  well  in  DDM  environments  people  typically 


Cue  integration  in  Dynamic  Decision  making 


Page  65 


HUMANS  STEMS 

(*t**p*i*  u4 


require  a  great  deal  of  experience  and  unambiguous  feedback;  however,  unambiguous  feedback  is 
seldom  available.  The  author  argues  that  this  is  why  learning  is  typically  difficult  and  performance 
poor  in  DDM.  Also  noted  is  the  importance  of  appropriate  mental  models  for  interpreting  feedback. 

The  difficulty  of  separating  inputs  and  outputs  in  DDM  is  mentioned.  People’s  difficulty  in 
incorporating  higher-order  effects  in  mental  models  is  a  problem  in  DDM  performance. 

Author  argues  for  a  dual-processing  framework,  where  intuitive  processing  is  primary  and 
analytical  processing  is  secondary,  but  often  concurrent. 

As  people  gain  experience,  there  is  a  shift  away  from  analytical  and  toward  intuitive  decision 
making  processes.  This  means  that  decisions  are  faster  but  also  more  prone  to  emotional  content 
and  other  biases. 

The  author  notes  that  most  DDM  experiments  are  done  with  novices,  and  so  might  not  capture  the 
behaviour  of  experts  in  a  DDM  domain.  A  key  defining  feature  of  decision-making  expertise  is 
knowing  when  NOT  to  trust  your  intuition  and  apply  more  rigorous  analytical  processes. 

The  author  discusses  several  approaches  to  training  in  a  DDM  environment,  depending  on  the  level 
of  expertise  desired  or  required  and  the  amount  of  time  available.  He  discusses  “high  road”  versus 
“low  road”  training,  and  uses  the  medical  profession  as  an  example.  A  “high  road”  approach  is 
geared  toward  producing  an  expert  surgeon,  requires  about  10  years  of  schooling  and  an  additional 
10  years  of  performing  under  supervision  before  true  mastery  is  obtained.  In  contrast,  a  “low  road” 
approach  is  similar  to  someone  learning  first  aid  -  very  basic  training  is  supplied,  but  the  person  is 
only  limited  in  competence.  As  the  amount  of  desired  skill  increases  and  the  amount  of  possible 
time  to  devote  to  learning  increases,  the  need  to  spend  time  initially  on  analytical-type  processing 
(which  will  gradually  become  intuitive  with  practice)  increases. 

The  author  notes  that  much  more  research  is  needed  to  determine  how  to  properly  design  a  “low 
road”  training  approach. 

This  is  an  interesting  article  which  points  out  that  perhaps  a  great  deal  of  expertise  is  not  always 
needed,  and  training  programs  should  take  into  account  the  level  of  expertise  needed  which  needs 
to  be  balanced  with  the  amount  of  time  which  can  be  devoted  to  training.  Important  limitations  in 
training  DDM  are  pointed  out.  However,  actual  applicable  solutions  are  not  really  offered;  rather 
the  author  just  points  to  more  research  being  needed  (although  a  reference  for  a  “high  road” 
method  is  offered). 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS  (?)  AND  BIASES 
To  be  used?  INCLUDE  IF  RELEVANT 


4.  Title:  Decisions  within  complex  systems:  An  experimental  approach  using  the  STRATEGEM-2 
computer  game 

Authors):  Bois,  J.  R. 

Source:  Dissertation  submitted  to  the  University  at  Albany,  State  University  of  New  York. 

Year:  2002 
Abstract: 

In  1989,  John  Sterman  published  his  seminal  paper.  Misperceptions  of  Feedback  in  Dynamic 
Decision  Making.  His  misperception  of  feedback  hypothesis  deals  with  the  difficulty  people  have 
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in  managing  complex  environments  even  when  they  purportedly  have  perfect  knowledge  and  have 
perfect  information  about  the  system.  Over  the  years,  several  authors  have  attempted  to  consider 
how  the  human  failures,  which  are  a  prominent  part  of  the  misperception  of  feedback  hypothesis, 
can  be  reduced.  However,  these  authors  have  achieved  mixed  results  in  attempting  to  make 
improvements  to  human  decision  support.  It  is  the  purpose  of  the  current  research  to  provide 
meaningful  decision  support  to  managers  of  complex  environments.  Specifically,  the  research  used 
the  STRATEGEM-2  simulation  game  and  purposely  developed  a  decision  support  method 
designed  to  improve  human  performance  within  a  complex  system.  The  experiment  required 
subjects  to  make  a  single  decision  within  a  dynamic  system  where  the  task  involved  feedback 
delays,  nonlinearity  of  system  processes,  positive  feedback  loops,  and  multiple  cues.  The  decision 
support  included  a  decision  rule  and  a  newly  developed  game  instruction  designed  to  improve 
participant  knowledge  and  information  about  the  microeconomy  of  the  STRATEGEM-2 
simulation.  Results  of  the  research  have  discovered  that  the  new  instruction  and  the  decision 
support  rule  produced  significant  results  in  improving  decision  making.  Additionally,  this  research 
demonstrates  that  the  lack  of  participant  motivation  levels  can  mask  decision  support  interventions. 
Subjects  with  high  self-assessed  motivation  outperformed  those  subjects  with  lesser  motivation 
levels. 

Notes: 

The  goal  of  this  paper  is  to  investigate  errors  in  DDM;  specifically,  why  errors  happen  even  though 
decision  makers  have  “perfect”  information  -  this  is  related  to  the  misperception  of  feedback  issue. 
The  overall  goal  is  to  determine  why  the  errors  happen  so  that  they  can  be  supported.  A  decision 
support  rule  was  used,  and  the  investigation  involved  determining  the  benefit  derived  from  the  use 
of  this  decision  aid.  Motivation  was  also  used  as  a  factor. 

There  is  a  discussion  about  the  “perfect  information”  assumption  -  often  participants  were  often 
not  tested  as  to  their  actual  information  level,  so  whether  or  not  participants  actually  understood  the 
cues  they  were  given  was  not  actually  known.  Testing  has  revealed  that  often  participants  don’t 
have  perfect  information,  and  their  performance  can  be  improved  by  improving  their 
understanding. 

Hypotheses  included: 

1 .  If  information  and  knowledge  about  a  system  are  better  understood,  participant  performance 
will  improve 

2.  If  participants  are  provided  with  a  decision  rule  that  focuses  their  attention  on  proper  cues  and 
how  to  weigh  their  importance,  their  performance  will  improve 

3.  Participants  reporting  greater  effort  during  the  experiment  simulation  will  outperform  those  that 
do  not  (p.  5) 

In  general,  the  hypotheses  were  supported. 

The  STRATEGEM  game  is  a  micro-economy  simulation.  The  player  acts  as  a  manager  for  a 
capital-producing  sector  of  an  economy.  Decisions  are  required  related  to  how  to  manipulate  goods 
and  capital  (processing  capabilities).  Complexity  is  added  such  that  changes  take  time  to  be 
implemented  and  to  affect  their  related  variables. 

Task  complexity  is  also  discussed.  Nine  variables  are  proposed  to  affect  task  complexity:  the 
number  of  variables,  interactions  between  subsystems,  random  variation,  miscellaneous  task 
characteristics  (a  catch-all  for  things  not  included  on  the  list),  time  delay  and  lagged  effects, 
effectiveness  of  decisions  on  outcomes  (i.e.,  ability  to  interpret  lagged  effects),  frequency  of 
oscillation  (stability  of  the  system),  positive  feedback  and  gains,  and  real-time  simulation. 
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Decision  support  was  given  in  2  forms.  The  first  was  the  Richardson  and  Rohrbaugh  decision  rule 
(a  strategy  for  making  decisions  in  the  game).  A  second  was  an  on-screen  tutorial  which  helped 
participants  understand  how  the  game  is  played  and  what  information  is  being  conveyed. 

It  is  an  interesting  question  for  DDM  training  to  investigate  whether  training  should  be  aimed  at 
whether  people  can  be  taught  or  shown  how  to  make  better  decisions.  It  appears  to  be  only 
peripherally  related  to  our  four  categories. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  CUE  INTEGRATION  AND/OR  PATTERN 
RECOGNITION 

To  be  used?  FOR  CONSIDERATION 


5.  Title:  Is  it  time  to  use  the  right  side  of  our  brain?  A  comparison  of  analytical  and  naturalistic 
decision  making  processes 

Author(s):  Bonifay,  B.  F. 

Source:  Paper  submitted  to  the  Faculty  of  the  Naval  War  College,  Newport,  R.  I. 

Year:  2000 
Abstract: 

As  commanders,  how  do  we  make  decisions?  In  scientific  theory,  there  are  two  methods  or  models 
for  the  decision-making  processes  -  the  Analytical  Decision-Making  (ADM)  model  and  the 
Naturalistic,  or  Recognitional,  Decision-Making  (NDM)  model.  The  time  one  has  to  make  a 
decision  within  a  certain  situation  will  influence  the  decision  making  process.  With  the  factor  of 
time  most  prevalent  in  this  situation,  a  greater  emphasis  should  be  placed  on  intuitive  decision¬ 
making  processes.  In  developing  intuition,  commanders  can  gain  a  supportive  experience  base  by 
immersing  themselves  in  numerous  decision-making  situations.  Over  time,  simulations  expand  a 
commander's  pattern  recognition  ability  thus  improving  his  intuitive  decision  making  skills.  In  a 
final  analysis,  the  operational  commander  makes  decisions  dependent  on  the  situation  he  is  facing. 

I  contend  that  commanders  of  today  and  future  commanders  need  to  understand  and  incorporate  the 
intuitive  Naturalistic  Decision-Making  process.  The  intuitive  decision-making  inherent  in  the 
Naturalistic  Decision-Making  process  provides  a  sound  basis  for  determining  a  proper  course  of 
action  for  a  given  situation.  This  paper  is  not  designed  to  espouse  that  analytical  decision-making 
processes  be  forgotten,  but  illustrates  that  every  decision  is  determined  by  a  situation. 

Notes: 

Junior  officers  are  taught  to  make  decisions  analytically  to  choose  the  best  COA.  The  author  notes 
that,  due  to  the  speed  that  decision  making  must  be  made  and  there  is  need  for  creativity  and 
intuition;  however,  these  have  not  only  not  been  taught,  but  have  been  squelched  in  favour  of 
analytical  processing.  The  author  intends  to  offer  recommendations  for  using  intuition  as  a 
decision-making  tool. 

The  author  points  out  the  two  main  decision-making  processes,  analytical  and  intuitive  (or 
naturalistic),  and  points  out  pros  and  cons  of  each.  Analytical  decision  making  is  favoured  when  an 
optimal  solution  is  desired,  and  it  tends  to  be  low  in  bias;  however,  this  method  of  decision  making 
is  cognitively  demanding  and  requires  a  lot  of  time.  It  is  also  difficult  to  use  in  a  DDM  situation 
where  the  situation  is  constantly  changing,  as  information  must  be  gathered  and  analysed,  and  the 
usefulness  of  that  information  is  fleeting  if  it  is  only  valid  for  a  little  time.  Discusses  naturalistic 
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decision  making  as  pattern  recognition.  Intuitive  decision  making  is  best  when  satisficing  is 
adequate  and  time  is  limited. 

The  author  points  out  that  both  types  of  decision  making  can  be  used  together;  however, 
naturalistic  decision  making  offers  speed,  which  is  its  major  advantage  over  analytical  processing. 

How  do  we  train  and  develop  intuition?  A  problem  is  that  intuition  is  not  typically  trusted  within 
the  military  culture.  The  author  suggests  using  an  approach  from  business  (pointing  out  that  the 
military  has  used  approaches  from  business  before).  Harvard  business  school  uses  an  approach  to 
teach  decision  making  which  uses  a  broad  base  of  cases  to  develop  an  understanding  of  the  best 
types  of  decisions.  The  importance  of  experience  is  emphasized. 

The  author  notes  the  irony  in  using  intuitive  decision  making.  It  is  most  necessary  when  speed  is  an 
issue;  this  means  that  lower-ranking  officers  are  more  likely  to  need  it.  However,  experience  is  the 
biggest  way  that  intuition  is  learned;  therefore,  people  who  are  most  skilled  (have  the  most 
experience)  are  generally  farthest  away  from  the  points  where  immediate  action  is  required. 

This  article  points  out  the  key  link  between  experience  and  intuitive  decision  making,  as  well  as  the 
times  when  intuitive  decision  making  is  most  needed  in  the  military  context.  It  also  clearly  points 
out  why  the  military  context  can  be  considered  a  DDM  environment. 

It  is  difficult  to  train  DDM  without  a  great  deal  of  experience;  experience  supplies  the  decision 
maker  with  a  knowledge  base  of  patterns  and  the  ability  to  recognize  those  patterns.  The  article  is 
better  at  pointing  out  the  need  for  training  intuitive  decision  making  rather  than  providing  concrete 
methods  for  doing  so  besides  presenting  people  with  a  large  number  of  case  studies  and  making 
sure  that  readers  attempt  to  understand  the  decisions  that  were  made,  how  they  were  made,  and  the 
relative  merits  of  those  decisions. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  PATTERN  RECOGNITION 
To  be  used?  INCLUDE  IF  RELEVANT 


6.  Title:  Use  of  experts  in  complex  decision  making:  A  paradigm  for  the  study  of  staff  work 
Author(s):  Brehmer,  B.  &  Hagafors,  R. 

Source:  Organizational  Behavior  and  Human  Decision  Processes,  38,  181-195. 

Year:  1986 
Abstract: 

Presents  a  general  experimental  paradigm  for  the  study  of  staff  decision  making  based  on  the  basic 
principles  of  social  judgment  theory.  The  paradigm  was  used  to  study  a  fundamental  problem  of 
staff  decision  making,  that  of  judging  what  weight  the  inputs  from  different  staff  members  should 
have  in  the  final  decision.  3  groups  of  10  high  school  students  performed  a  fictitious  medical  task 
that  required  them  to  learn  to  make  judgments  about  the  severity  of  a  disease  on  the  basis  of  6  cues. 
Ss  had  access  to  3  experts  that  integrated  information  from  2  cues.  Results  show  that  Ss  were  able 
to  learn  in  all  3  conditions,  but  learning  was  better  in  the  equal  validity/equal  ability  condition  than 
in  conditions  in  which  validity  and  ability  differed.  Differences  in  using  the  information  obtained 
from  the  experts  and  cues  across  conditions  indicate  the  difficulty  in  assigning  weights  in  a 
judgment  task. 

Notes: 
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The  authors  note  that  complexity  is  often  a  problem  in  decision  making;  one  way  of  dealing  with 
this  is  to  divide  the  problem  into  subproblems,  and  assign  these  subproblems  to  different  experts.  A 
central  decision  maker  then  has  to  integrate  the  information  from  the  experts  and  arrive  at  a 
decision.  One  problem  is  that  the  decision  maker  has  to  decide  how  much  weight  to  give  to 
information  about  different  subproblems  (from  different  experts).  Two  relevant  aspects  to 
weighting  are  the  judgment  of  the  expert  (e.g.,  how  much  the  information  can  be  trusted),  and  how 
important  the  subproblem  is  to  the  overall  problem. 

The  purpose  of  this  paper  is  to  suggest  a  paradigm  in  which  staff  decision-making  can  be 
investigated.  A  preliminary  experiment  was  performed  which  investigated  whether  a  decision 
maker  is  able  to  learn  to  give  different  weights  to  experts  whose  accuracy  differs.  Note  that  the 
participants  were  high-school  students  making  decisions  about  a  fictitious  medical  task.  The  task 
used  nonsense  names  for  cues  and  a  nonsense  disease. 

The  results  indicate  that  participants  are  sensitive  to  the  validity  of  cues  and  expert  validity; 
evidence  was  mixed  about  which  set  of  criteria  were  weighted  more  heavily  and  what  factors 
influenced  this  weighting. 

This  task  is  relevant  to  DDM  because  information  complexity  is  one  of  the  major  issues  with 
DDM;  as  well,  the  approach  taken  here,  of  dividing  a  main  problem  into  subproblems,  is  one  of  the 
strategies  used  in  military  DDM.  Unfortunately,  conclusions  cannot  be  derived  from  this  work  as 
the  experimental  results  are  preliminary  and  not  straightforward;  as  well,  participants  were  not 
experts. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  CUE 
INTEGRATION _ 

To  be  used?  INCLUDE  IF  RELEVANT 


7.  Title:  Risk  Perception  and  Strategic  Decision  Making:  General  Insights,  a  New  Framework,  and 
Specific  Application  to  Electricity  Generation  Using  Nuclear  Energy. 

Authors):  Brewer,  J.  D. 

Source:  Report  prepared  for  Sandia  National  Laboratories,  Albuquerque,  NM,  Sandia  Report 
Number  SAND2005-5730. _ 

Year:  2005 _ 

Abstract: 

The  objective  of  this  report  is  to  promote  increased  understanding  of  decision  making  processes 
and  hopefully  to  enable  improved  decision  making  regarding  high-consequence,  highly 
sophisticated  technological  systems.  This  report  brings  together  insights  regarding  risk  perception 
and  decision  making  across  domains  ranging  from  nuclear  power  technology  safety,  cognitive 
psychology,  economics,  science  education,  public  policy,  and  neural  science  (to  name  a  few).  It 
forms  them  into  a  unique,  coherent,  concise  framework,  and  list  of  strategies  to  aid  in  decision 
making.  It  is  suggested  that  all  decision  makers,  whether  ordinary  citizens,  academics,  or  political 
leaders,  ought  to  cultivate  their  abilities  to  separate  the  wheat  from  the  chaff  in  these  types  of 
decision  making  instances.  The  wheat  includes  proper  data  sources  and  helpful  human  decision 
making  heuristics;  these  should  be  sought.  The  chaff  includes  “unhelpful  biases”  that  hinder  proper 
interpretation  of  available  data  and  lead  people  unwittingly  toward  inappropriate  decision  making 
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“strategies”;  obviously,  these  should  be  avoided.  It  is  further  proposed  that  successfully 
accomplishing  the  wheat  vs.  chaff  separation  is  very  difficult,  yet  tenable.  This  report  hopes  to 
expose  and  facilitate  navigation  away  from  decision-making  traps  which  often  ensnare  the  unwary. 
Furthermore,  it  is  emphasized  that  one’s  personal  decision  making  biases  can  be  examined,  and 
tools  can  be  provided  allowing  better  means  to  generate,  evaluate,  and  select  among  decision 
options.  Many  examples  in  this  report  are  tailored  to  the  energy  domain  (esp.  nuclear  power  for 
electricity  generation).  The  decision  making  framework  and  approach  presented  here  are  applicable 
to  any  high-consequence,  highly  sophisticated  technological  system. 

Notes: 

The  author  intends  to  provide  an  overview  of  decision-making  processes  and  improve  decision 
making  in  DDM-type  contexts  (i.e.,  complex  decisions  involving  complex  technologies  in  complex 
domains).  The  author  tries  to  create  a  coherent  and  concise  framework  by  combining  information 
about  risk  perception  and  decision  making  across  many  domains.  There  is  discussion  of  relevant 
factors  for  understanding  decision  types,  including  level  of  care,  consequence  of  failure,  and 
tolerance  of  failure. 

The  author  talks  about  “appropriate”  and  “inappropriate”  strategies;  however,  the  context  is 
generally  what  makes  a  heuristic  appropriate  or  not,  and  the  author  incorrectly  implies  that  it  is  the 
heuristics  themselves  that  should  be  used  or  avoided.  The  author  also  notes  that  his  method  applies 
when  “time  is  available  for  thoughtful  reflection”;  therefore  it  may  not  be  appropriate  for  many 
DDM  domains  with  high  stress  and  time  pressure. 

There  is  discussion  of  uncertainty  of  information.  The  article  contains  a  large  list  of  normative 
biases  (e.g.,  insensitivity  to  sample  size),  heuristics  (e.g.,  anchoring),  and  individual-specific  biases 
(e.g.,  loss  aversion).  Note  that  the  author’s  understanding  of  heuristics  may  be  incomplete  -  he 
seems  to  be  confusing  the  availability  and  representativeness  heuristics.  The  author  also  tends  to 
represent  heuristics  as  always  negative. 

There  are  discussions  which  may  prove  useful,  including  a  discussion  of  “critical  thinking  skills” 
which  may  help  avoid  bias,  and  a  discussion  of  bias  related  to  “spiral  of  stereotypes”,  which  results 
in  distrust  between  groups  with  different  sets  of  knowledge/experience,  resulting  in  accepted 
beliefs  in  each  group  being  perceived  as  more  credible  and  members  of  a  group  that  support  the 
other  group  being  considered  irrational. 

The  author  performs  a  useful  review  of  heuristics,  biases,  and  other  issues  that  can  make  decision 
making  “irrational”;  however,  discussions  of  how  to  deal  with  these  challenges  are  of  limited  use 
for  our  DDM  context  as  the  mitigation  strategies  require  significant  amounts  of  time  to  implement, 
plus  significant  amounts  of  time  to  train.  As  well,  they  are  not  “natural”,  meaning  that  under  time 
pressure  and  stress  they  are  likely  to  be  poorly  implemented. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  HEURISTICS,  BIASES 
To  be  used?  FOR  CONSIDERATION 


8.  Title:  The  use  of  recognition  information  and  additional  cues  in  inferences  from  memory 
Author(s):  Broder,  A.  &  Eichler,  A. 

Source:  Acta  Psychologica  121, 275-284 
Year:  2006 
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Abstract: 

Goldstein  and  Gigerenzer’s  (2002)  [Goldstein,  D.  G.  &  Gigerenzer,  G.  (2002).  Models  of 
ecological  rationality:  The  recognition  heuristic.  Psychological  Review,  109,  75-90]  “Recognition 
Heuristic”  (RH)  was  tested  for  its  empirical  validity  in  an  experimental  paradigm  with  induced 
recognition  of  objects.  RH  claims  that  upon  inferring  which  of  two  objects  (e.g.,  cities)  scores 
higher  on  a  criterion  (e.g.,  city  size),  a  recognized  object  will  be  chosen  over  an  unrecognized  one, 
if  the  recognition  is  a  valid  predictor  of  the  criterion  without  considering  additional  object 
information.  Trying  to  avoid  potential  shortcomings  of  former  studies,  we  (a)  used  the  city 
population  task,  (b)  provided  additional  cue  information  only  for  recognized  cities,  and  (c)  had 
participants  draw  inferences  from  memory.  Participants  learned  city  names  and  additional 
information  about  some  cities.  They  also  learned  that  recognition  and  the  additional  information 
were  valid  predictors  of  the  criterion  “city  size”.  In  a  subsequent  decision  phase,  the  additional 
information  about  the  cities  in  memory  strongly  affected  the  inferences,  suggesting  that  recognition 
information  is  clearly  integrated  into  judgments,  but  by  no  means  in  a  noncompensatory  fashion 
that  would  dominate  every  other  cue. 

Notes: 

Broder  and  Eichler  test  the  empirical  validity  of  the  “recognition  heuristics”  (RH)  proposed  by 
Goldstein  and  Gigerenzer  (2002).  Empirical  support  for  this  theory  is  lacking  so  they  did  a  study. 
They  look  at  the  claims  that  RH  is  used  in  a  noncompensatory  nature,  meaning  that  the  information 
of  recognition  is  always  used  if  it’s  available,  and  used  exclusively. 

The  authors  did  a  study  that  used  the  “city  size”  paradigm.  Participants  were  to  choose  between 
two  city  names  to  identify  which  had  a  larger  population.  The  test  used  had  two  conditions.  One, 
participants  learned  names  of  cities,  and  the  second,  participants  learned  the  names  of  cities  and 
additional  positive  (e.g.,  the  city  does  have  a  soccer  stadium)  or  negative  (e.g.,  the  city  does  not 
have  a  soccer  stadium)  information  about  the  city.  Participants  were  then  tested  to  see  if  they  would 
use  the  additional  information  in  order  to  deduce  which  city  was  larger,  or  if  recognition 
information  was  used  in  a  noncompensatory  manner. 

It  was  found  that  people  integrate  recognition  information  into  their  judgements  showing  that 
participants  will  choose  recognized  cities  more  often  than  those  not  recognized.  Results  showed 
that  other  cue  information  was  not  ignored,  but  rather  incorporated  into  the  decisions  made.  The 
additional  information  largely  determined  the  choice  regardless  if  the  other  city  in  the  pair  was 
recognized  or  new.  Interestingly,  recognition  information  seemed  to  be  dominant  in  that  using  cue 
information  of  a  recognized  object  in  making  the  decision  does  show  a  noncompensatory  use  of 
RH.  They  argue  then  that  the  claim  that  RH  is  noncompensatory  in  nature  is  at  least  questionable. 

This  article  identifies  that  recognition  information  is  dominant  which  can  lead  us  to  investigate  this 
type  of  information.  They  also  find  that  cue  information  can  be  integrated  as  other  information  is 
used  in  DM,  rather  than  to  be  cast  aside  in  light  of  recognition  information  as  proposed  by  previous 
researchers. 

They  were  unable  to  provide  evidence  that  disproved  the  noncompensatory  nature  of  RH,  however 
conclude  that  it  is  still  questionable. 

Challenges  of  using  heuristics  were  not  explicitly  discussed,  however,  the  literature  discussed 
suggests  that  people  have  difficulty  using  information  inappropriate  way.  When  too  much 
information  is  given  within  a  particular  decision  it  can  degrade  their  responses,  rather  than  those 
who  make  a  similar  decision  with  less  information,  as  in  the  ‘ Hess  is  more  ”  idea. 

There  seems  to  be  difficulty  in  using  cue  information  in  conjunction  with  recognized  information. 


Page  72 


Cue  integration  in  Dynamic  Decision  making 


Humansys/e/ws 


HUMAN  SYSTEMS 


There  are  biases  in  using  the  information  appropriately,  thus  cognitive  challenges  may  be  in 
discriminating  and  appropriately  weighting  information  in  DM. 

Note  that  this  article  uses  simple  decision  making  environments. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS _ 

To  be  used?  INCLUDE  IF  RELEVANT 


9.  Title:  Training  effects  on  the  use  of  simple  heuristics  in  threat  assessment 
Author(s):  Bryant,  D.  J. 

Source:  Report  prepared  for  Defence  Research  &  Development  Canada,  DRDC-Toronto  Report 
No.  TR-2005-229. _ 

Year:  2005 _ 

Abstract: 

Previous  research  has  examined  the  use  of  fast  and  frugal  heuristics  for  threat  classification  with 
probabilistic  cues.  In  all  previous  studies  subjects  learned  the  underlying  relationships  of  cues  to 
friend/foe  classification  through  trial-and-error  learning.  Explicit  training,  however,  is  theoretically 
interesting  because  it  potentially  involves  the  integration  of  deliberate  cognitive  learning  of  the 
task’s  underlying  cue  structure  with  implicit,  procedural  learning  of  specific  cue-criterion 
probabilities.  In  the  experiment,  subjects  learned  to  classify  contacts  in  a  simulated  naval  warfare 
environment  and  then  were  tested  on  sets  of  contacts  that  were  designed  to  contrast  predictions  of 
several  heuristics,  including  the  Take-the-Best-for-  Classification  (TTB-C)  and  Pros  Rule 
developed  specifically  for  the  threat  classification  task,  as  well  as  a  Bayesian  strategy  based  on 
computation  of  the  conditional  probabilities  of  friend  or  foe  classification  given  the  particular 
pattern  of  cues.  The  interpretability  of  the  results  was  limited  by  the  large  proportion  of  subjects 
who  exhibited  uninterpretable  patterns  of  responding.  In  contrast  to  previous  experiments,  very  few 
subjects  employed  TTB-C,  although  more  did  use  the  less  frugal  Pros  Rule.  The  experiment  did 
yield  the  novel  finding  that  some  subjects  can,  given  explicit  training,  employ  a  Bayesian  strategy 
for  this  task. 

Notes: 

The  goal  of  this  paper  was  to  investigate  the  cognitive  processes  underlying  threat  assessment,  and 
determine  if  explicit  training  could  improve  performance.  Threat  assessment  involves  the 
interpretation  and  integration  of  probabilistic  cues,  due  to  the  nature  of  the  information  received. 

Pattern-based  (typical  patterns  of  cues  are  learned)  and  cue-based  (diagnostic  information  about 
each  cue  is  provided)  explicit  methods  of  training  are  compared.  It  was  proposed  by  the  author  that 
pattern-based  training  was  more  consistent  with  a  Baysian  strategy  (where  all  cues  are  used  and 
weighted)  and  cue-based  training  was  more  consistent  with  heuristics  that  use  the  most  diagnostic 
cue.  Overall  performance  seemed  slightly  better  overall  when  participants  received  cue-based 
training. 

Several  strategies  are  investigated,  including  “take  the  best”,  which  means  using  the  single  most 
valid  cue  to  determine  the  response.  This  strategy  tends  to  increase  in  frequency  of  use  as  the  cost 
of  cue  information  increases.  Another  is  a  Pros  Rule,  which  is  a  procedure  which  includes 
calculating  the  sum  of  cue  values  for  each  alternative  and  selecting  the  alternative  with  the  highest 
score.  These  Pros  rules  can  either  be  weighted  or  unweighted. 
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Unfortunately,  many  of  the  participants  didn’t  produce  a  pattern  of  results  that  were  clearly 
classifiable  into  the  expected  strategies,  making  interpretation  of  the  results  difficult.  This  limits  the 
usefulness  of  this  article.  There  are  some  hints  about  the  impact  of  different  training  types  on 
decision  making  accuracy,  and  one  interesting  finding  is  that  in  this  explicit  training  condition, 
participants  tended  to  use  more  resource-intensive  strategies  (which  possibly  could  improve 
accuracy).  Thus,  the  training  may  not  have  resulted  in  the  use  of  the  decision  method  predicted. 

This  article  is  not  really  a  DDM  scenario;  however,  it  could  probably  be  expanded  to  apply. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION, 
AND/OR  PATTERN  RECOGNITION 

To  be  used?  INCLUDE  IF  RELEVANT 


10.  Title:  Tools  for  understanding  and  aiding  dynamic  decision  making 
Author(s):  Busemeyer,  J.  R.  &  Pleskak,  T.  J. 

Source:  Dissertation  presented  to  Michigan  State  University  (note  -  this  originally  appeared  in 
search  as  a  Journal  of  Mathematical  Psychology  article) 

Year:  2008 _ 

Abstract: 

Dynamic  decisions  arise  in  many  applications  including  military,  medical,  management,  sports,  and 
emergency  situations.  During  the  past  50  years,  a  variety  of  general  and  powerful  tools  have 
emerged  for  understanding,  analyzing,  and  aiding  humans  faced  with  these  decisions.  These  tools 
include  expected  and  multi-attribute  utility  analyses,  game  theory,  Bayesian  inference  and  Bayes 
nets,  decision  trees  and  influence  diagrams,  stochastic  optimal  control  theory,  partially  observable 
Markov  decision  processes,  neural  networks  and  reinforcement  learning  models,  Markov  logics, 
and  rule-based  cognitive  architectures.  What  are  all  of  these  tools,  how  are  they  related,  when  are 
they  most  useful,  and  do  these  tools  match  the  way  humans  make  decisions?  We  address  all  of 
these  questions  within  a  broad  overview  that  is  written  for  an  interdisciplinary  audience.  Each 
description  of  a  tool  introduces  the  principles  upon  which  it  is  based,  and  also  reviews  empirical 
research  designed  to  test  whether  humans  actually  use  these  principles  to  make  decisions.  We 
conclude  with  suggestions  for  future  directions  in  research. 

Notes: 

The  authors  note  that  many  tools  for  understanding  decision  making  seem  to  fall  into  at  least  two 
groups:  tools  with  which  to  model  the  decision  situation  (e.g.,  decision  trees,  influence  diagrams) 
and  some  are  tools  that  can  be  used  to  analyze  situations  and  come  to  a  (sometimes  optimal) 
decision. 

The  author  provides  a  framework  for  understanding  DDM,  including  a  choice,  time  points,  outputs, 
influences  on  the  environment  from  decisions,  and  uncertainty. 

The  authors  describe  three  main  frameworks  for  modelling  human  learning  processes  in  DDM: 
production  rule  models  (e.g.,  ACT-R)  which  assume  that  a  large  set  of  condition-action  rules  are 
incrementally  learned  from  experience;  exemplar  (or  case-based  or  instance)  learning,  which 
assume  whenever  an  action  leads  to  a  successful  outcome  the  situation  and  successful  response  are 
stored  together  in  memory,  with  retrieval  based  on  similarity  to  the  situation;  and  supervised  neural 
network  models  which  use  feedback  to  adjust  weights  which  map  inputs  to  outputs. 
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There  is  a  discussion  at  the  end  of  the  article  about  the  assumptions  of  various  models  which  are 
violated  (e.g.,  that  backward  induction  plays  a  major  role  in  decision-making,  an  assumption  of 
many  models  which  is  not  supported  by  human  data). 

This  article  is  heavily  mathematical,  and  so  it  is  difficult  to  understand  how  this  could  be  very 
usefully  applied  to  real-world  DDM  problems  with  uncertainty  and  time  pressure.  It  seems  more 
geared  toward  understanding  general  DDM  rather  than  ways  of  specifically  improving  DDM,  and 
also  seems  to  focus  on  utility  functions  which  other  articles  have  described  as  not  being  suitable  for 
describing  actual  human  performance.  The  article  does  make  an  interesting  point,  which  is  that 
decision  making  in  DDM  should  primarily  aim  for  robustness  (e.g.,  being  suitable  across  a  wide 
variety  of  options  for  the  way  a  system  could  evolve)  than  for  optimality  based  on  the  current  or 
predicted  future  situation. 

This  article  mentions  heuristics,  biases,  and  many  of  the  models  are  related  to  cue  integration  and 
pattern  recognition;  however,  this  article  is  not  clearly  geared  toward  any  of  those  and  explores 
DDM  in  general. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  CUE 
INTEGRATION,  AND/OR  PATTERN  RECOGNITION 

To  be  used?  FOR  CONSIDERATION 


1 1 .  Title:  The  Accuracy  of  Intelligence  Assessment:  Bias,  Perception,  and  Judgement  in  Analysis 
and  Decision 

Author(s):  Butterfield,  A.  P.  Jr. 

Source:  Paper  submitted  to  the  Naval  War  College,  Newport,  R.  I. 

Year:  1993 
Abstract: 

This  paper  examines  bias,  perception,  and  judgment  in  intelligence  analysis  and  decision  making.  It 
asks  if  the  accuracy  of  intelligence  assessments  can  be  improved.  An  answer  is  sought  at 
fundamental  levels  of  analysis,  where  biases  influence  observation,  observations  are  mediated  by 
preconception,  and  perceptions  pass  through  the  filter  of  critical  judgment.  Unintentional  human 
errors  in  observing,  perceiving,  and  judging  are  the  central  issues  of  this  paper.  Deliberate 
distortions  of  intelligence  through  political  calculation  or  service  parochialism  are  treated 
peripherally.  A  survey  of  literature  leads  to  the  conclusion  that  analytic  bias  is  inevitable;  that 
intelligence  concepts  are  necessary  and  dangerous;  and  that  uncertainty  ensures  a  margin  of  error  in 
assessment.  Therefore,  efforts  to  eliminate  bias  and  increase  the  objectivity  of  intelligence  officers 
are  unproductive.  Instead,  efforts  should  focus  on  the  suitability  and  adaptability  of  concepts. 
Intelligence  officers  can  be  encouraged  to  use  concepts  flexibly  by  exercising  several  qualities  of 
critical  judgment....  Intelligence,  Misperception  of  Adversaries,  Bias  and  Intelligence,  Perception 
and  Intelligence,  Judgment  and  intelligence,  Ethics  and  intelligence.  Analysis  and  Decision 

Notes: 

The  author  examines  bias  in  intelligence  analysis.  One  conclusion  is  that  attempts  to  remove  bias 
will  be  unproductive,  and  so  biases  must  be  examined  and  understood,  and  attempts  made  to 
educate  people  about  them  and  how  they  can  be  appropriate  and  inappropriate. 

The  author  mentions  that  mistakes  in  intelligence  analysis  can  be  the  result  of  unintentional  bias, 
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but  purposeful  deception  can  also  play  a  role. 

The  author  specifies  three  levels  of  intelligence  assessment: 

•  Seeing  and  observing  -  drawing  analytic  inferences  from  raw  intelligence  reports  (problem 
of  bias) 

•  Observing  and  believing  -  mediation  of  observations  by  the  beliefs  and  expectations  of  the 
analyst  (problem  of  perception  and  misperception) 

•  Believing  and  thinking  -  filtering  perceptions  through  the  screen  of  reason  before  making  a 
decision  (problem  of  judgment  and  ethical  thinking) 

By  organizing  intelligence  analysis  in  this  way,  the  author  hopes  to  shed  light  on  types  of  problems 
which  are  often  encountered  but  not  made  explicit. 

Intelligence  analysis  inevitably  contains  subjective  inferences,  but  conventionally  intelligence 
analysis  is  supposed  to  be  objective.  It  is  impossible  in  intelligence  analysis  to  usefully  separate 
qualitative  and  quantitative  information;  e.g.,  to  separate  assessments  of  capabilities  from 
assessments  of  intention. 

Distortions  can  take  many  forms,  including  misunderstanding  clear  signals,  undermining  or 
enhancing  the  validity  of  information,  and  confirmation  bias.  The  author  notes  that  one  reason  for 
the  fact  that  there  are  many  examples  of  misperceptions  in  wartime  is  that  crisis  increases 
ambiguity,  and  preconceptions  are  often  used  to  deal  with  ambiguity,  leading  to  misperceptions. 
Intelligence  must  be  structured;  facts  to  be  included  must  be  selected  for  intel  reports. 

The  author  notes  several  attempted  methods  for  dealing  with  ambiguity  and  bias.  The  scientific 
approach  of  quantifying  analysis  and  decision  making  imparts  an  “artificial  order”  on  an  inherently 
disorderly  process.  Bureaucratic  reforms  such  as  encouraging  multiple  viewpoints  involve  tradeoffs 
such  as  decreasing  preconceptions  but  increasing  ambiguity. 

The  author  suggests  that  a  better  approach  to  dealing  with  bias  is  to  discuss  which  biases  are  best 
and  which  concepts  are  right  for  a  given  situation.  “Intelligence  aims  at  success,  not  the  truth”  (p. 
viii).  Factors  that  should  be  considered  include  tolerance  for  error,  requirements  for  sensitivity  to 
change,  and  degree  of  ambiguity. 

Five  qualities  of  critical  judgment  can  be  used  to  improve  decision  making  p.  x): 

•  Full  awareness  of  the  inherent  limits  of  intelligence 

•  Explicitness  of  assumptions  and  procedures 

•  Deliberate  analytic  self-consciousness 

•  Intuition  supported  by  knowledge 

•  Honesty  and  integrity  in  analysis  and  in  the  relationship  between  intelligence  advisors  and 
decision-makers 

This  article  suggests  that,  as  the  use  of  heuristics  and  biases  are  unavoidable,  that  people  who  have 
to  make  decisions  must  understand  how  they  affect  reasoning  and  what  factors  are  important  when 
determining  which  types  of  reasoning  to  use.  This  is  more  of  a  discussion  than  a  concrete  method 
of  improving  performance  in  DDM. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS  and  BIASES 
To  be  used?  INCLUDE  IF  RELEVANT 
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1 2.  Title:  Training  Decision  Makers  for  Complex  Environments:  Implications  of  the  Naturalistic 
Decision  Making  perspective 

Author(s):  Canon-Bowers,  J.  A.  &  Bell,  H.H. 

Source:  In  Naturalistic  Decision  Making.  C.E.  Zsambok  &  G.  Klein  (Eds.),  pp.  99-1 10.  Mahwah 
NJ:  Lawrence  Erlbaum  Associates. 

Year:  1997 
Abstract: 

The  purpose  of  this  chapter  is  to  examine  Naturalistic  Decision  Making  (NDM)-related  theorizing 
from  the  standpoint  of  what  it  implies  for  how  we  train  decision  makers.  It  is  based  largely  on  the 
panel  discussion  (from  the  Second  Naturalistic  Decision  Making  conference  held  in  Dayton  OH  in 
June  1994)  devoted  to  this  topic.  This  chapter  focuses  on  generating  propositions  for  training  the 
cognitive  aspects  of  decision  making  as  suggested  by  the  NDM  perspective.  Suggestions  are 
offered  for  future  research  in  this  area. 

Notes:  This  paper  is  based  on  a  panel  discussion  devoted  to  NDM  theories  for  the  purpose  of 
identifying  implications  for  training  decision  makers.  Panellists  included:  John  Schmitt,  Hugh 
Wood,  Rannan  Lipshitz  and  Jon  Fallesen. 

The  authors  identified  1)  characteristics  of  effective  decision  makers,  2)  mechanisms  of  expert 
decision  making  and  3)  implications  for  how  to  train  decision  makers. 

Characteristics  of  effective  decision  makers  included: 

•  Flexible 

•  Quick 

•  Resilient 

•  Adaptive 

•  Risk  taking 

•  Accurate 

Mechanisms  of  expert  decision  making  (from  which  implications  for  training  can  be  identified) 
include: 

•  Situation  assessment  skills  -  includes  cue  recognition  and  pattern  recognition 

•  Organized  knowledge  structures  -  includes  building  knowledge  templates  and  matching 
them  to  new  situation 

•  Mental  simulation  -  simulation  of  potential  solutions  for  novel  situations 

•  Strategy  selection  -  need  to  continually  assess  and  modulate  strategy;  requires 
metacognition 

•  Reasoning  skills  -  includes  causal  reasoning,  hypothesis  generation,  hypothesis  testing, 
using  analogies,  critical  thinking  skills  and  domain-specific  problem-solving  skills 

Implications  of  NDM  theories  for  training  include: 

•  What  to  train  -  metacognitive  skills,  reasoning  skills,  domain-specific  problem-solving 
skills,  mental  simulation  skills,  risk  assessment  skills,  situation  assessment  skills, 
knowledge  organization 
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•  How  to  train  -  simulations,  guided  practice  and  feedback,  embedded/organic  training, 
cognitive  apprenticeships,  multi-media  presentation  formats 

Article  assessment: 

Not  based  on  empirical  investigations  for  NDM-based  training  interventions  because  very  few,  if 
any,  exist. 

Good  base  for  future  investigations  especially  related  to  the  impact  of  training  decision  makers  in 
NDM-based  strategies/techniques. 

Primary  training  challenge  is  that  NDM  is  inherently  not  analytical  and  therefore  does  not  use  a 
structured  process  so  is  therefore  difficult  to  train.  Training  therefore  focuses  on  the  skills  required 
to  make  effective  decisions  in  a  DDM  environment. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  PATTERN  RECOGNITION,  CUE 
RECOGNITION,  CUE  INTEGRATION  AND  ALSO  TRAINING  IMPLICATIONS  FOR 
NDM/DDM 

To  be  used?  INCLUDE  IF  RELEVANT 


13.  Title:  Training  battlefield  critical  thinking  and  initiative 

Author(s):  Cohen,  M.  S.,  Thompson,  B.  B.,  Adelman,  L.,  &  Bresnick,  T.  A. 

Source:  Interim  report  submitted  to  the  U.  A.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences 

Year:  1999 _ 

Abstract: 

The  first  objective  of  this  research  was  to  explore  and  identify  the  cognitive  skills  that  individuals 
need  to  function  effectively  in  domains  that  require  them  to  cope  with  uncertainty,  change  and 
conflicting  purposes.  The  second  objective  was  to  develop  and  test  methods  for  training  those  skills 
in  the  context  of  Army  battlefield  decision  making.  The  training  aimed  at  improving  the  ability  of 
Army  tactical  staff  officers  to  grasp  the  essential  elements  of  a  complex,  uncertain,  and  dynamic 
situation,  visualize  those  elements  in  terms  of  their  organizational  goals,  and  take  action  in  a  timely 
and  decisive  manner.  The  identification  of  cognitive  skills  to  be  trained  and  training  strategies  is 
based  on  an  original  theory  of  critical  thinking  skills,  the  Recognition/Metacognition  (R/M) 

Theory.  The  theoretical  and  empirical  foundations  of  the  R/M  model  are  described  using  literature 
on  mental  models,  expertise,  behavioral  decision-making,  and  pattern  recognition.  The  critical 
thinking  skills  developed  for  training  include  (1)  developing  and  using  appropriate  mental  models 
of  high-order  purpose  and  of  time  orientation,  and  (2)  applying  critical  thinking  strategies  to  these 
models,  including  the  identification  and  filling  of  critical  information  gaps  and  conflicts  in  situation 
understanding,  goals  and  plans,  and  identifying  and  evaluating  underlying  plan  assumptions. 
Training  is  implemented  in  a  stand-alone  Compact  Disk  (CD). 

Notes: 

Focus  on  initiative  and  coordination  -  importance  of  both  and  how  to  balance  them.  Coordination 
(i.e.,  having  multiple  people  work  on  a  task)  has  benefits  but  tends  to  reduce  efficiency  unless  the 
task  is  made  less  flexible  (thereby  lessening  ability  to  have/show  initiative).  Coordination  can  also 
result  in  gains  through  multiple  points  of  view;  however,  groups  are  prone  to  bias  such  as _ 
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groupthink.  The  authors  propose  that  decentralization  is  one  way  of  handling  problems  with  group 
bias  and  loss  of  efficiency.  Properly  using  initiative  relies  upon  using  appropriate  critical  thinking 
about  mental  models  of  the  task  and  team. 

The  authors  argue  that  optimal  performance  in  DDM  contexts  rely  upon  appropriate  mental  models 
about  organizational  purpose,  own  and  others’  intent,  degrees  of  initiative,  team  member  reliability, 
and  action  sequences.  To  make  the  best  use  of  this  information,  the  ability  to  think  critically  about 
these  mental  models  and  apply  them  to  novel  situations  is  also  required. 

Discusses  pattern  recognition  as  one  type  of  expertise;  however,  the  ability  to  purposely  create 
novel  solutions  is  also  another  characteristic  of  expertise.  An  important  ability  of  experts  related  to 
pattern  recognition  is  the  ability  to  recognize  exceptions. 

According  to  the  authors,  a  fundamental  skill  is  related  to  separating  grounds  (what  is  given)  from 
conclusions  (what  is  inferred).  The  relationship  between  grounds  and  conclusions  are  not  generally 
1 00%  certain,  so  this  is  a  way  of  identifying  assumptions  which  might  have  to  be  tested. 

Focus  of  training  is  the  development  of  a  moderately-sized  set  of  mental  model  types  (e.g., 
purpose,  intent,  team  member  reliability)  and  critical  thinking  strategies  that  should  be  used  to 
critique  and  correct  those  mental  models  when  pattern  recognition  is  inadequate  (i.e.,  novel 
situations). 

This  is  an  interesting  article,  and  presents  a  way  of  thinking  about  decision  making  in  dynamic 
environments  as  a  mixture  of  pattern  recognition  and  the  use  of  general  mental  models  and  critical 
thinking.  Pattern  recognition  is  used  when  possible,  and  general  mental  models  and  critical 
thinking  are  the  best  things  to  use  when  dealing  with  novel  situations.  It  isn’t  quite  clear  how  a 
decision  maker  will  make  the  call  to  use  one  or  the  other  type  of  decision  making,  and  real  decision 
making  is  likely  to  be  a  combination  of  both  types.  The  training  information  is  interesting,  and  it 
may  result  in  an  improvement  in  decision  making,  but  it  is  not  clear  how  useful  this  method  will  be 
in  situations  with  tight  timelines  (i.e.,  when  people  are  likely  to  revert  to  the  most  natural  method 
of  processing  information  and  have  limited  time  to  perform  the  critical  thinking  component  of  the 
strategy). 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS  (MENTAL  MODELS), 
PATTERN  RECOGNITION 

To  be  used?  INCLUDE  IF  RELEVANT 


14.  Title:  Anchoring-and-adjustment  in  software  project  management:  An  experimental 
investigation 

Author(s):  Costello,  T.  P. 

Source:  Thesis  submitted  to  the  Naval  Postgraduate  School,  Monterey,  CA. 

Year:  1992 
Abstract: 

Software  project  development  continues  to  be  characterized  by  cost  overruns,  late  deliveries,  poor 
reliability  and  user  dissatisfaction.  The  Systems  Dynamics  Model  of  Software  Project  Management 
is  a  quantitative  model  of  software  project  dynamics  that  is  attempting  to  gain  some  valuable 
insight  into  the  managerial  side  of  developing  software  systems.  The  objective  of  this  thesis  was  to 
use  the  Systems  Dynamics  Model's  gaming  interface  to  investigate  the  cognitive  heuristic _ 
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anchoring-and-adjustment  in  dynamic  decision  environments,  and  its  use  in  software  project 
management.  Specifically,  subjects  were  provided  with  either  a  low  or  a  high  anchor  condition  to 
determine  the  effect  on  subject  productivity  estimation  and  project  performance  when  confronted 
with  dynamic  decision  making  in  software  project  management.  The  results  show  that  subjects 
used  anchoring  to  simplify  decision  making  in  the  complex  dynamic  environment.  There  was 
evidence  of  bias  introduced  by  the  anchor,  thereby  causing  dysfunctional  performance. 

Notes: 

Software  project  management  is  described,  and  appears  to  be  a  DDM  environment.  The  Systems 
Dynamics  Model  (SDM)  is  a  quantitative  simulation  of  software  project  dynamics,  and  has  a 
gaming  interface  which  enables  direct  interaction  with  the  simulation. 

Anchoring-and-adjustment  (A&A)  is  described  as  a  heuristic  whereby  an  initial  starting  point 
influences  the  final  estimate  toward  itself.  The  author  discusses  the  fact  that  A&A  in  a  dynamic 
environment  could  lead  to  an  accurate  estimate  due  to  continuous  processing,  at  least  if  feedback 
were  allowed  to  play  a  role  in  estimating. 

In  the  experiment,  participants  were  attempting  to  manage  a  software  development  project;  one 
group  was  given  a  low  productivity  estimate  for  workers  (the  low  anchor  condition)  and  the  other 
group  was  given  a  higher  productivity  estimate  for  workers  (the  high  anchor  condition).  It  is 
interesting  to  note  that,  even  in  the  low  productivity  estimate  condition,  participants  tended  to  treat 
the  original  estimate  as  optimistic,  and  the  first  revision  was  for  lower  expected  productivity  in 
both  anchor  conditions.  Results  indicate  that  productivity  estimates  made  in  the  high  anchor 
condition  were  significantly  higher  overall  than  productivity  estimates  made  in  the  lower  anchor 
condition,  indicating  that  the  anchor  had  an  effect  on  estimates  which  persisted  throughout  the 
experiment.  From  the  data  (and  the  nonsignificant  effect  of  time  and  time  by  anchor  and  time  by 
anchor  interactions),  it  appears  as  though  the  size  of  the  effect  did  not  greatly  diminish  over  time, 
as  would  be  expected  from  the  hypothesis  that  in  a  dynamic  environment  with  feedback  an 
anchor’s  effect  would  diminish  over  the  course  of  interacting  with  the  environment. 

Overall  performance  (i.e.,  the  completion  time  for  the  simulation  in  days)  for  the  more  difficult 
project  was  better  for  the  high  anchor  group,  but  there  was  no  significant  difference  between 
anchor  conditions  for  the  easier  project. 

One  interesting  effect  relevant  for  training  is  that  performance  appeared  to  be  better  when  the 
easier  of  the  2  projects  was  managed  first;  this  supports  a  training  method  which  increases  in 
difficulty. 

This  experiment  is  an  experiment  using  an  actual  DDM  context  which  allows  interaction  and 
concrete  evaluation  of  performance.  It  shows  an  effect  of  the  A&A  heuristic,  situations  under 
which  it  may  have  a  greater  effect,  and  some  factors  which  may  influence  how  A&A  affects  overall 
performance.  The  impact  of  training  order  on  difficult  versus  easy  simulations  is  also  relevant. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES 
To  be  used?  INCLUDE  IF  RELEVANT 


1 5.  Title:  Impact  of  diffusion  and  variability  on  vendor  performance  evaluation 
Author(s):  Doerr,  K.,  Eaton,  D.R.,  &  Lewis,  I. 

Source:  Report  prepared  for  the  Naval  Postgraduate  School,  Monterey,  CA.  Report  No.  NPS  LM- 
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Year:  2005 
Abstract: 

In  this  paper  we  develop  a  theory  of  the  impact  of  behavioral  decision  making  factors  on  the 
evaluation  of  logistic  service  providers  under  performance-based  logistics  and  provide  an  analysis 
of  pilot  data  collected  in  an  attempt  to  find  support  for  that  theory.  Based  on  a  review  of  the  logistic 
measurement,  PBL,  and  behavioral  decision  making  literature,  we  form  four  hypotheses  about 
specific  impacts  of  process  measures  and  variance  on  performance  evaluation  in  PBL.  Our  first 
hypothesis  is  that  the  difficulty  of  relating  component-level  measures  to  system-level  outcomes 
will  lead  to  an  increased  use  of  non-diagnostic  or  only  partially  diagnostic  process  measures.  We 
further  propose  that  these  process  measures  will  produce  a  dilution  effect  in  which  system 
outcomes  are  undervalued.  Our  third  hypothesis  is  that  absent  clear,  observable  outcome  metrics  at 
the  component  level,  decision  makers  will  increasingly  rely  on  measures  of  inputs  as  surrogates  for 
outputs.  Our  fourth  hypothesis  is  that  absent  a  specific  guidance  on  how  to  value  variance,  decision 
makers  will  tend  to  overlook  this  important  component  of  performance.  We  report  results  from  a 
pilot  test  conducted  to  develop  an  instrument  that  will  be  used  to  try  to  find  support  for  hypotheses 
two  and  four. 

Notes: 

For  weapons  systems,  logistics  are  often  not  considered  as  part  of  the  evaluation  process  of  the 
outcome  (e.g.,  an  assessment  of  whether  weapons  are  available  does  not  often  have  a  direct  place  in 
the  evaluation  process  of  a  successful  mission).  Instead  biases  ensue  and  among  others,  the  dilution 
effect  occurs  (non-diagnostic  information  causes  diagnostic  information  to  be  undervalued). 

Performance-based  logistics  investigated  in  this  article  is  concerned  with  the  measurement  of 
performance  data  of  vendors  providing  weapons  system  logistics.  The  authors  seek  to  explore  the 
role  that  select  biases  (e.g.  outcome  bias,  input  bias,  curse  of  knowledge,  dilution  effect)  could 
have  in  this  new  process.  Hypotheses  are  made  about  what  effect  biases  will  have  and  then  a  study 
is  conducted  to  explore  these  hypotheses.  Mixed  support  is  found  for  some  of  the  hypotheses.  A 
short  discussion  is  provided  that  states  how  the  biases  described  may  be  implemented  in  the  PBL. 

There  are  strong  limitations  to  this  report.  The  idea  that  biases  and  heuristics  are  used  is  confirmed 
through  the  pilot  test.  Though  the  biases  are  briefly  described,  there  are  no  details  about  the  origins 
of  the  biases  or  evidence  of  their  use  presented.  The  list  of  biases  chosen  seems  rather  haphazard, 
with  no  explanation  on  what  some  biases  are  included  or  excluded  from  the  discussion.  The  results 
are  mixed  (at  best). 

There  is  no  DDM  environment  and,  no  learning  opportunities  presented. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  BIASES 
To  be  used?  DO  NOT  INCLUDE 


16.  Title:  Cognitive  adaptation  and  its  consequences:  A  test  of  cognitive  continuum  theory 
Author(s):  Dunwoody,  P.  T.,  Haarbauer,  E.,  Mahan,  R.  P.,  Marino,  C.,  &  Tang,  C.-C. 
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Abstract: 

Cognitive  Continuum  Theory  (CCT)  is  an  adaptive  theory  of  human  judgement  and  posits  a 
continuum  of  cognitive  modes  anchored  by  intuition  and  analysis.  The  theory  specifies  surface  and 
depth  task  characteristics  that  are  likely  to  induce  cognitive  modes  at  different  points  along  the 
cognitive  continuum.  The  current  study  manipulated  both  the  surface  (information  representation) 
and  depth  (task  structure)  characteristics  of  a  multiple-cue  integration  threat  assessment  task.  The 
surface  manipulation  influenced  cognitive  mode  in  the  predicted  direction  with  an  iconic 
information  display  inducing  a  more  intuitive  mode  than  a  numeric  information  display.  The  depth 
manipulation  influenced  cognitive  mode  in  a  pattern  not  predicted  by  CCT.  Results  indicate  this 
difference  was  due  to  a  combination  of  task  complexity  and  participant  satisficing.  As  predicted, 
analysis  produced  a  more  leptokurtic  error  distribution  than  intuition.  Task  achievement  was  a 
function  of  the  extent  to  which  participants  demonstrated  an  analytic  cognitive  mode  index,  and  not 
a  function  of  correspondence,  as  predicted.  This  difference  was  likely  due  to  the  quantitative  nature 
of  the  task  manipulations. " 

Notes: 

The  authors  propose  that  most  decision  making  is  a  mix  of  analytical  and  intuitive  processes;  what 
they  call  “quasi-rational”  decision  making.  The  position  the  authors  take  is  that  neither  analytical  or 
intuitive  decision  making  are  inherently  better;  what  makes  one  or  the  other  more  appropriate  has  a 
lot  to  do  with  the  nature  of  the  task  (e.g.,  whether  rational  for  decisions  will  have  to  be  made),  the 
nature  of  the  inputs  (e.g.,  whether  cues  should  be  weighted  equally;  how  many  cues  must  be 
integrated),  and  other  factors.  They  also  indicate  that  the  way  information  is  given  can  influence 
where  on  the  “cognitive  continuum”  decision  making  will  occur. 

One  interesting  point  is  that  analytical  decision  making  does  not  necessarily  provide  a  ceiling  for 
performance;  one  reason  is  that  errors  made  during  analytical  decision  making  tend  to  have  more 
profound  effects  on  the  outcome  than  do  errors  made  during  intuitive  decision  making.  In  other 
words,  errors  may  be  more  frequent  in  intuitive  decision  making,  but  the  answer  is  likely  to  be 
closer  to  the  true  answer  than  when  errors  occur  in  analytical  decision  making.  Thus,  intuitive 
decision  making  can  be  more  robust. 

The  paper  describes  an  experiment  which  is  intended  to  test  the  cognitive  continuum  theory  by 
inducing  different  types  of  decision  making  in  participants.  They  found  that  an  iconic  information 
display  induced  a  more  intuitive  decision  style  than  a  numeric  information  display.  Unlike  their 
predictions,  they  found  a  nonlinear  relationship  between  the  task  characteristics  and  the  actual 
decision  making  style  selected  by  participants.  One  possible  reason  why  this  might  have  occurred 
is  that  the  same  decision  had  to  be  made  in  all  conditions.  As  well,  participants  may  have  been 
driven  to  reduce  cognitive  effort,  and  so  been  more  likely  to  choose  an  intuitive  strategy  no  matter 
the  form  of  the  task  characteristics.  Thus,  the  authors  state  that  “close  correspondence  between  task 
characteristics  and  cognitive  characteristics  may  be  less  important  than  the  overall  functionally 
adaptive  value  of  quasi-rational  cognition  when  confronting  a  task  with  high  analytic  depth  features 
that  includes  a  high  degree  of  environmental  predictability.”  (p.  50). 

This  article  is  relevant  for  DDM,  as  it  is  important  for  decision  makers  to  apply  different  decision 
strategies  depending  on  the  task  demands,  although  the  task  used  here  is  not  strictly  DDM. 
Understanding  how  best  to  train  decision  making  in  DDM  should  take  account  the  weaknesses 
inherent  in  analytical  reasoning,  including  the  one  pointed  out  here;  that  errors  can  be  more 
profound  in  analytical  reasoning.  It  is  also  important  to  note  that  the  way  information  is  displayed 
will  influence  the  style  of  decision  making  chosen;  this  may  also  mean  that  the  way  information  is 
displayed  during  training  could  skew  the  decision  maker  to  a  particular  decision  style  which  may 
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persevere  even  when  it  is  no  longer  suitable,  although  this  hypothesis  should  be  examined. 

The  article  does  not  really  address  the  influence  of  individual  differences,  which  likely  also  have  a 
profound  impact  on  the  type  of  decision  making  strategy  used. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION 
To  be  used?  INCLUDE  IF  RELEVANT 


17.  Title:  Multiple  goals  in  dynamic  decision  making:  An  experimental  approach 
Author(s):  Elser,  E.  T. 

Source:  Thesis  submitted  to  the  Naval  Postgraduate  School,  Monterey,  CA. 

Year:  1993 
Abstract: 

Leaders  in  both  the  military  and  civilian  sectors  make  a  series  of  interrelated  decisions  in  real  time 
to  achieve  goals.  These  decisions  involve  the  allocation  of  resources,  such  as  ships  and  aircraft  to 
influence  the  situation  facing  the  decision  maker.  NEWFIRE  is  a  computer-based  simulation  of  a 
forest  fire  fighting  task  that  allows  the  experimenter  to  control  both  the  goals  and  the  environment 
in  which  the  decisions  are  made  and  thereby  explore  the  effects  these  variables  have  on  the 
decision  maker.  The  objective  of  this  thesis  was  to  use  the  NEWFIRE  microworld  to  determine  the 
effects  that  multiple  goals  and  system  complexity  have  on  decisions.  Specifically,  subjects  were 
given  one,  two  or  three  goals,  and  confronted  with  three  scenarios  of  varying  complexity.  The 
results  show  that  subjects  given  only  one  objective  outperformed  those  given  two  or  three 
objectives.  The  results  also  show  that  the  performance  of  subjects  on  the  most  complex  scenario 
was  worse  than  on  the  less  complex  scenarios. 

Notes: 

The  main  goals  of  this  paper  was  to  investigate  the  role  that  goal  conflict  (number  of  goals)  affects 
the  quality  of  decision  making  in  DDM  and  how  task  complexity  affects  this  relationship.  The 
methodology  involves  using  a  simulated  microworld  involving  firefighting,  which  allows  a 
concrete  investigation  of  DDM  including  concrete  goals  and  measureable  objectives  which  could 
be  useful  for  training. 

The  hypotheses  investigated  were  that  participants  will  perform  better  without  goal  conflict,  that 
participants  will  perform  more  poorly  as  goal  conflict  increases  (i.e.,  from  two  to  three  goals),  and 
that  people  will  perform  more  poorly  as  task  complexity  increases.  The  justification  for  this 
research  was  that  goal  conflict  has  seldom  actually  been  investigated  in  DDM. 

Hypotheses  were  supported  in  general,  although  there  were  some  interesting  (and  unpredicted) 
interactions  between  task  complexity  and  goal  complexity.  There  was  a  suggestion  that  the  three 
goal  (i.e.,  most  complex  goal  condition)  group  outperformed  the  two  goal  group  on  the  easy  and 
moderately  difficult  scenarios,  but  performed  the  worst  in  the  difficult  scenario.  The  author 
suggests  that  the  three-goal  group  might  have  been  more  efficient  in  goal  shifting  than  the  two-goal 
group. 

The  hypotheses  investigated  in  this  article  are  largely  very  intuitive  and  so  the  results  are  not 
surprising.  The  one  exception  is  the  interaction  between  goal  and  task  complexity.  It  could  be  an 
interesting  area  for  research  to  determine  the  limits  to  the  benefits  of  goal  shifting,  and  determine 
the  cons  as  well  (e.g.,  this  might  result  in  more  burnout  as  it  requires  more  resources,  so  people 
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could  not  perform  the  task  for  as  long  as  they  could  with  a  more  simple  goal  structure). 

Note  that  it  is  not  entirely  clear  whether  this  paper  supports  one  or  more  of  the  four  topics  of 
interest,  although  it  could  possibly  support  cue  integration  and  pattern  recognition  in  a  broad  sense. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  CUE  INTEGRATION,  PATTERN 
RECOGNITION? _ 

To  be  used?  FOR  CONSIDERATION  -  some  interesting  ideas  but  relevance  not  certain 


18.  Title:  Subjective  assessment  of  uncertainty 
Author(s):  Fallon,  R. 

Source:  Paper  submitted  to  the  Rand  Graduate  Institute,  Santa  Monica,  CA. 

Year:  1976 _ 

Abstract: 

Properly  assessing  your  subjective  probabilities  is  extremely  difficult,  and  is  more  of  an  art  than  a 
science.  One's  subjective  probability  of  an  uncertain  event  should  truly  reflect  his  proper  state  of 
information  if  it  is  to  be  of  any  value  at  all  for  planning  purposes.  In  this  study  the  author  points  out 
many  of  the  common  cognitive  influences,  and  biases  which  they  lead,  that  people  undergo,  either 
consciously  or  unconsciously,  in  the  intuitive  assessment  of  probability.  These  heuristics  and  biases 
are  not  only  prevalent  in  people  of  a  variety  of  backgrounds  but  are  also  common  in  experienced 
researchers  they  think  intuitively.  A  better  understanding  of  these  simple  heuristics,  and  the  biases 
to  which  they  lead,  by  improving  the  ability  for  individuals  to  properly  assess  their  true  state  of 
information,  could  improve  judgements  and  decisions  in  situations  of  uncertainty.  It  is  very 
important,  therefore,  that  advisers  and  analysts  who  are  involved  in  planning  for  uncertain  future 
events  be  aware  of,  and  consciously  try  to  avoid,  these  influences. 

Notes: 

This  paper  reviews  several  heuristics  and  the  biases  which  they  tend  to  produce,  including 
anchoring  and  adjustment,  availability,  and  representativeness.  People’s  experience  with  statistics 
and  their  familiarity  with  making  probabilistic  judgments  do  not  seem  to  significantly  affect  their 
use  of  heuristics  and  bias  that  accompanies  them. 

The  quality  of  this  paper  is  in  some  doubt,  as  it  was  prepared  for  a  graduate  course  and  is  fairly  old. 
It  primarily  discusses  the  use  of  heuristics,  and  does  not  necessarily  relate  to  DDM  or  how  to  train 
people  to  properly  use  heuristics  and  avoid  bias. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES _ 
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1 9.  Title:  Models  of  Information  Aggregation  Pertaining  to  Combat  Identification:  A  Review  of  the 
Literature 

Author(s):  Famewo,  J.,  Matthews,  M.,  &  Lamoureux,  T. 
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Year:  2007 
Abstract: 

This  report  reviews  research  literature  pertaining  to  models  of  human  information  aggregation  in 
order  to  apply  them,  when  possible,  to  combat  identification  (CID)  decisions  made  by  individuals 
in  a  land  force  context.  In  particular,  the  review  examined  social  science,  science  and  military 
literature  to  identify  principles  and  theories  associated  with  how  human  decision  makers  weigh  and 
integrate  information.  The  review  identified  a  variety  of  heuristics  and  cognitive  biases  associated 
with  these  processes,  along  with  formal  and  informal  models  of  combining  qualitative  and 
quantitative  information.  The  report  presents  current  conceptualizations  of  combat  identification 
and  places  it  within  the  context  of  cue-based  decision  making  under  the  intuitive  decision 
framework.  By  framing  this  complex  decision  we  were  able  to  evaluate  the  applicability  of  the 
methods  of  aggregation  to  the  CID  task. 

Notes: 

The  purpose  of  the  report  was  to  identify  plausible  human  decision  making  models,  approaches  and 
limitations  in  aggregating  information  for  the  purposes  of  combat  identification  (CID).  Key 
researchers  in  the  decision  making,  heuristics  and  biases  area  (Amos  Tversky  at  Stanford 
University,  Kahneman  at  Princeton  University;  Gerd  Gigerenzer  and  Daniel  G.  Goldstein  at  Mac 
Planck  Institute  for  Human  Development)  as  well  as  those  in  Cue-based  Decision  Making  (Gary 
Klein  at  Klein  and  Associates;  Raanan  Lipshitz  at  Haifa  University;  and  Eduardo  Salas  at  the 
University  of  Central  Florida)  are  briefly  included. 

The  authors  conclude  that  research  into  natural  decision  making  (including  Recognition  Primed 
Decision  Making  and  the  Brunswickian  Lens  Model)  as  well  as  behavioural  decision  theory 
(including  classic  heuristics  and  biases,  as  well  as  fast  and  frugal  heuristics)  are  both  more  useful 
for  describing  the  CID  process  than  rational  decision  making  models.  Each  of  these  models,  along 
with  the  use  of  cues,  patterns,  heuristics  and  biases  are  discussed.  The  role  of  uncertainty  in 
decision  making  was  also  included.  Two  methods  for  overcoming  uncertainty  specifically 
mentioned  were:  the  Reducing,  Assumption,  Weighting,  Forestalling  and  Suppressing  (RAWFS) 
heuristic;  and,  Recognition/Meta-Cognition  (R/M)  model. 

Specific  cognitive  challenges  in  CID  decision  making  noted  were: 
improper  weighting  of  cues  due  to  various  circumstances 
classic  limits  of  working  memory  (7+-2  units) 

decision  variability  based  on  the  presentation  of  information  (e.g.  order  effects,  salience) 
overconfidence,  confirmatory,  and  ignoring  of  base  rates  biases 
adjustment  and  anchoring;  take-the-best  (and  other  fast  and  frugal)  heuristics 
herding  behaviour  effects 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  CUES,  PATTERN  RECOGNITION, 
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Author(s):  Garcia-Retamero,  R.,  Hoffrage,  U.,  &  Dieckmann,  A. 

Source:  The  Quarterly  Journal  of  Experimental  Psychology,  60,  1197-1215. 

Year:  2006 _ 

Abstract: 

One-reason  decision-making  heuristics  as  proposed  by  Gigerenzer,  Todd,  and  the  ABC  Research 
Group  (1999)  have  been  shown  to  perform  accurately.  However,  such  strategies  cannot  deal  with 
compound  cues.  We  propose  the  Take  The  Best  Configural  Cue  (TTB-Configural)  as  a  fast  and 
frugal  heuristic  that  processes  compound  cues.  In  a  series  of  three  experiments,  we  analysed 
whether  participants  used  this  heuristic  when  making  cue-based  inferences  on  which  of  two 
alternatives  had  a  higher  criterion  value.  In  two  of  the  experiments,  two  cues  were  amalgamated 
into  a  valid  compound  cue  by  applying  the  AND  or  the  OR  logical  rule,  respectively.  In  the  third 
experiment,  there  was  no  valid  compound  cue.  Within  each  experiment,  we  also  manipulated 
causal  mental  models  through  instructions.  In  the  configural  causal  model,  cues  were  said  to  act 
through  the  same  causal  mechanism.  In  the  elemental  causal  model,  cues  were  said  to  act  through 
different  causal  mechanisms.  In  the  neutral  causal  model,  the  causal  mechanism  was  not  specified. 
When  a  highly  valid  compound  existed,  and  participants  had  a  configural  causal  model,  for  the 
majority  of  them  the  strategy  that  could  best  account  for  their  choices  was  TTB-Configural. 
Otherwise,  the  strategy  that  best  predicted  their  choices  was  the  Take  The  Best  (TTB)  heuristic. 

Notes: 

The  authors  propose  a  method  of  processing  compound  cues  termed  “Take  The  Best  Configural 
Cue”  (TTB-configural)  as  a  fast  and  frugal  heuristic.  The  TTB  is  a  heuristic  designed  to  infer 
which  of  two  alternatives  has  a  higher  value  on  a  quantitative  criterion.  The  TTB  uses  validity 
(probability  that  a  particular  cue  will  lead  to  a  correct  decision).  There  is  no  integration  of 
information  as  the  decision  is  based  on  the  first  discriminating  cue.  The  TTB-Configural  starts  with 
this  theory  and  integrates  the  use  of  compound  cues.  It  starts  by  searching  for  a  particular  set  of 
cues,  including  both  compound  and  individual  cues. 

The  authors  conduct  three  experiments  in  which  they  analyze  if  participants  use  this  heuristic  when 
making  cue-based  inferences.  This  type  of  heuristic  may  be  used  when  searching  for  information  in 
the  environment  is  costly  or  when  decisions  have  to  be  made  under  time  pressure. 

Previous  experiments  showed  that  when  participants  were  instructed  to  search  for  highly  valid 
compound  cues,  and  these  existed,  the  TTB-configural  strategy  was  the  best  fit  for  the  strategies 
used  (based  on  expected  time  to  use  this  strategy).  When  instructed  that  cues  act  through  different 
causal  mechanism  or  participants  had  no  info  about  causal  mechanisms  TTB  was  used  (a  one- 
reason  decision-making  heuristic  that  deals  with  non-compound  cues). 

Experiment  1  manipulated  participant’s  “mental  models”  of  the  environment.  The  groups  were 
configural  (instruction  encouraged  participants  to  search  for  highly  valid  compound  cues  in  the 
environment),  elemental  (instruction  emphasized  that  cues  act  through  different  causal  mechanisms 
when  determining  the  criterion;  this  was  meant  to  encourage  participants  to  treat  cues 
independently),  and  neutral  (no  information  about  possible  causal  mechanisms  through  which  cues 
acted)  causal  models.  Experiment  2  replicated  results  found  in  Experiment  1  and  used  a  higher 
payoff  as  incentive  for  participants.  Experiment  3  contained  no  valid  compound  cue  (compound 
AB  =  0.5  corresponding  to  guessing).  All  other  aspects  of  the  experiment  were  identical  to  1  &  2. 

Results  showed  that  when  participants  were  given  instruction  that  cues  acted  through  the  same 
causal  mechanism  the  majority  of  participants  made  decisions  best  modelled  by  the  TTB- 
Configural  heuristic.  When  instructions  suggested  cues  were  acting  independently  their  decisions 
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were  best  modeled  by  the  elemental  TTB.  This  was  also  the  case  for  the  neutral  causal  model 
condition.  In  an  environment  with  no  compound  cues,  no  participants  were  categorized  as  using 
TTB-Configural,  but  instead  were  modelled  best  by  the  TTB.  Those  in  the  configural  condition 
looked  up  more  information  during  the  experiment  as  they  may  have  been  more  uncomfortable 
making  the  decision  due  to  their  original  “mental  model”  that  did  not  prove  useful  in  an 
environment  with  no  compound  cues. 

These  studies  provide  evidence  that  the  TTB-Configural  heuristic,  constructed  from  building 
blocks  of  TTB,  can  be  created  as  an  adaptive  tool  in  decision  making.  Future  research  may 
investigate  what  happens  when  imperfect  validity  relationships  occur  between  the  compound  cues 
and  the  outcome  and  what  results  occur  when  a  substantial  increase  in  compound  cues  are 
presented  in  the  environment. 

The  authors  demonstrate  a  number  of  scenarios  in  which  participants  strategies  are  best  reflected 
by  fast  and  frugal  heuristics  (take-the-best  heuristic,  and  TTB-configural  heuristic).  The  show  that 
building  on  a  more  simplistic  heuristic,  adaptive  decision  making  tools  can  be  formed  when  more 
complex  decision  making  is  required. 

This  study  also  demonstrates  that  confidence  in  decision  making,  and  comfort  level  with  the 
amount,  and  quality  of  the  information  seems  to  be  an  influencing  factor  in  determining  whether  to 
search  for  more  information  in  attempts  to  use  incorporating  more  analytic  strategy,  or  simply  use 
fast  and  frugal  heuristics  with  minimal  information. 

Results  from  the  studies  indicate  that  when  information  cues  are  have  a  more  direct,  causal 
relationship  to  the  criterion,  strategies  used  implied  a  type  of  “mental  model”  (what  the  authors 
define  as  a  mental  representation  of  the  problem  solving  strategy  structure),  that  gather  and  process 
individual  pieces  of  information  as  well  as  compound  pieces  of  information  (combinations  of 
cues),  rather  than  only  individual  cues. 

Note  that  within  these  studies  decision  tasks  were  not  dynamic. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION 
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2 1 .  Title:  The  power  of  causal  beliefs  and  conflicting  evidence  on  causal  judgements  and  decision 
making 

Author(s):  Garcia-Retamero,  R.,  Mueller,  S.  M.,  Catena,  A.,  &  Maldonado,  A. 
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Abstract: 

In  two  experiments,  we  investigated  the  relative  impact  of  causal  beliefs  and  empirical  evidence  on 
both  decision  making  and  causal  judgments,  and  whether  this  relative  impact  could  be  altered  by 
previous  experience.  Participants  had  to  decide  which  of  two  alternatives  would  attain  a  higher 
outcome  on  the  basis  of  four  cues.  After  completing  the  decision  task,  they  were  asked  to  estimate 
to  what  extent  each  cue  was  a  reliable  cause  of  the  outcome.  Participants  were  provided  with 
instructions  that  causally  related  two  of  the  cues  to  the  outcome,  whereas  they  received  neutral 
information  about  the  other  two  cues.  Two  of  the  four  cues — a  causal  and  a  neutral  cue — had  high 
validity  and  were  both  generative.  The  remaining  two  cues  had  low  validity,  and  were  generative  in 
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Experiment  1,  but  almost  not  related  to  the  outcome  in  Experiment  2.  Selected  groups  of 
participants  in  both  experiments  received  pre-training  with  either  causal  or  neutral  cues,  or  no  pre¬ 
training  was  provided.  Results  revealed  that  the  impact  of  causal  beliefs  and  empirical  evidence 
depends  on  both  the  experienced  pre-training  and  cue  validity.  When  all  cues  were  generative  and 
participants  received  pre-training  with  causal  cues,  they  mostly  relied  on  their  causal  beliefs, 
whereas  they  relied  on  empirical  evidence  when  they  received  pre-training  with  neutral  cues.  In 
contrast,  when  some  of  the  cues  were  almost  not  related  to  the  outcome,  participants’  responses 
were  primarily  influenced  by  validity  and — to  a  lesser  extent — by  causal  beliefs.  In  either  case, 
however,  the  influence  of  causal  beliefs  was  higher  in  causal  judgments  than  in  decision  making. 
While  current  theoretical  approaches  in  causal  learning  focus  either  on  the  effect  of  causal  beliefs 
or  empirical  evidence,  the  present  research  shows  that  both  factors  are  required  to  explain  the 
flexibility  involved  in  human  inferences. 

Notes: 

Two  experiments  are  conducted  that  investigate  the  impact  of  causal  beliefs  and  empirical  evidence 
on  decision  making  and  causal  judgements.  The  studies  also  examined  if  previous  experience  could 
alter  the  relative  impact  between  these.  Specifically,  they  looked  at  rational  decision  making  and 
decisions  related  to  the  capacity  to  apply  inferences  about  the  future  drawn  from  experiences  that 
verify  or  falsify  previous  causal  beliefs. 

The  first  study  examines  how  much  individuals  rely  on  previous  causal  beliefs  when  interpreting 
empirical  evidence.  Results  conclude  participants’  decisions  were  guided  by  causal  beliefs  and  by 
empirical  evidence  (cue  validity).  Absence  of  pre-training  and  pre-training  with  causal  cues  lead  to 
a  clear  preference  for  causal  cues.  Pre-training  with  valid  cues  that  were  not  causally  linked  to  the 
outcome  did  not  have  any  influence  on  participants’  decisions  when  presented  with  causal 
instruction  about  the  cues  (results  independent  of  causal  status  of  cues).  The  second  experiment 
aimed  to  replicate  findings  in  Experiment  1  and  further  investigate  if  causal  judgements  and 
decision  making  tap  into  different  psychological  mechanisms  as  no  correlation  was  observed 
between  these  2  variables  in  Experiment  1 . 

Subjecting  participants  to  causal  beliefs  did  impact  their  causal  judgements.  Differences  in  groups 
showed  evidence  that  decisions  and  causal  judgements  tap  different  psychological  processes. 
Overall,  both  decision  making  and  causal  judgments  were  primarily  influenced  by  cue  validity  (the 
degree  to  which  the  cue  actually  contributes  to  the  outcome),  and  to  a  lesser  extent  by  preconceived 
causal  beliefs.  Results  show  participants  that  were  exposed  to  causal  cues  (both  with  and  without 
pre-training)  mainly  relied  on  high-validity  cues  that  were  causally  linked  to  the  outcome  and 
disregarded  causal  low-validity  cues  (not  related  to  the  outcome).  Participants  exposed  to  neutral 
cues  prior  to  the  trials  (with  and  without  pre-training)  relied  preferentially  on  high-validity  cues 
regardless  of  whether  they  were  causal  or  neutral.  This  proposes  that  causal  beliefs  have  a  great 
impact  on  causal  judgements.  This  is  in  line  with  research  that  describes  confirmation  bias. 

Future  direction:  the  authors  suggest  looking  at  whether  there  is  a  way  that  those  exposed  to 
preconceived  causal  beliefs  can  become  more  sensitive  to  validity  information  rather  than 
maintaining  focus  on  the  causal  cues.  The  authors  discuss  a  Belief  Revision  Model  and  a  causal 
belief  -  reliability  equation  that  examine  how  causal  beliefs  and  empirical  information  may  impact 
the  interpretation  of  data  and  consequently  the  impact  on  a  decision  or  judgement. 

The  authors  point  out  that  there  may  be  different  processes  used  in  making  different  types  of 
decisions  and  judgements.  This  may  impact  further  investigation  of  DDM  and  training,  in  that  a 
each  type  of  process  of  decision  making  may  need  to  be  investigated  individually,  if  indeed  there 
are  large  differences. 
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Cue  validity  was  pointed  out  as  one  of  the  primary  influencing  factors  in  making  inferences.  The 
validity  of  cues  is  subjective  within  DDM,  thus  this  may  be  one  major  challenge  in  performing 
DDM  tasks. 

If  people  are  able  to  have  reliable  high  validity  cues,  they  are  less  likely  to  rely  on  prior  causal 
beliefs.  This  implies  that  fewer  biases  will  impact  the  decision  and  perhaps  even  heuristics  would 
be  used  when  there  is  high  validity  of  cues.  The  previous  Garcia-  Retamero  et  al.  paper  (2006) 
indicates  that  people’s  confidence  in  the  quality  (recency,  accuracy)  of  the  cues  may  impact  the  use 
of  valid  cues.  The  challenges  in  this  situation  may  be  more  environmental  rather  than  cognitive. 

Pre-established  beliefs  were  found  to  influence  causal  judgements  more  than  decision  making. 
However,  biases  and  previous  experience,  as  they  may  be  based  on  pre-established  beliefs  (and 
help  to  establish  beliefs),  we  understand  from  other  literature,  can  severely  influence  decision 
making. 

This  article  does  not  directly  focus  on  particular  biases  or  heuristics,  though  we  may  understand 
beliefs  in  a  way  that  may  act  similarly  in  decision  making. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  related  to  BIASES,  HEURISTICS _ 
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22.  Title:  Does  causal  knowledge  help  us  be  faster  and  more  frugal  in  our  decisions? 

Author(s):  Garcia-Retamero,  R.,  Wallin,  A.,  &  Dieckmann,  A. 

Source:  Memory  &  Cognition,  35,  1399-1409 

Year:  2007 

Abstract: 

One  challenge  that  has  to  be  addressed  by  the  fast  and  frugal  heuristics  program  is  how  people 
manage  to  select,  from  the  abundance  of  cues  that  exist  in  the  environment,  those  to  rely  on  when 
making  decisions.  We  hypothesize  that  causal  knowledge  helps  people  target  particular  cues  and 
estimate  their  validities.  This  hypothesis  was  tested  in  three  experiments.  Results  show  that  when 
causal  information  about  some  cues  was  available  (Experiment  1),  participants  preferred  to  search 
for  these  cues  first  and  to  base  their  decisions  on  them.  When  allowed  to  learn  cue  validities  in 
addition  to  causal  information  (Experiment  2),  participants  also  became  more  frugal  (i.e.,  they 
searched  fewer  of  the  available  cues),  made  more  accurate  decisions,  and  were  more  precise  in 
estimating  cue  validities  than  was  a  control  group  that  did  not  receive  causal  information.  These 
results  can  be  attributed  to  the  causal  relation  between  the  cues  and  the  criterion,  rather  than  to 
greater  saliency  of  the  causal  cues  (Experiment  3).  Overall,  our  results  support  the  hypothesis  that 
causal  knowledge  aids  in  the  learning  of  cue  validities  and  is  treated  as  a  meta-cue  for  identifying 
highly  valid  cues. 

Notes: 

The  authors  examine  the  relationship  between  causal  information  and  the  use  of  cues  to  guide 
decision  making.  Their  hypotheses  include  the  idea  that  causal  information  should  act  as  a  guide  to 
determine  which  cues  to  examine  to  uncover  their  validity,  and  also  help  to  remember  the  validity 
of  those  cues.  In  effect,  causal  information  acts  as  a  “meta-cue”,  or  a  cue  about  the  cues,  in 
determining  what  is  helpful.  This  is  important  because  in  many  contexts,  including  DDM,  there  is 
an  overabundance  of  cues,  little  information  (at  least  at  first)  about  which  cues  are  valid,  and 
decision  makers  have  limited  working  memory  with  which  to  learn  the  relationships  between  the 
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cues  and  outcomes  and  which  cues  are  most  valid.  In  fact,  without  explicit  information  about  cue 
validity,  the  authors  point  out  that  decision  makers  don’t  actually  seem  to  remember  information 
about  cue  validity. 

The  authors  find  that  the  addition  of  causal  information  influences  the  cues  which  participants 
choose  to  examine  (the  causal  ones  over  noncausal  cues),  increases  the  speed  with  which  they 
make  decisions,  increases  the  accuracy  of  those  decisions,  increases  confidence  in  decisions,  and 
increases  the  accuracy  of  estimations  of  cue  validity. 

This  topic  seems  very  relevant  to  decision  making  in  DDM,  as  it  is  a  major  question  how  people 
select  and  integrate  cues,  and  how  this  can  be  done  quickly  and  accurately.  Understanding  the  role 
of  causal  information  helps  to  explain  and  reduce  some  of  the  cognitive  complexity  inherent  in 
DDM.  This  may  also  have  a  beneficial  impact  on  training,  as  pointing  out  causal  information  could 
make  it  easier  for  people  to  remember,  as  well  as  learn  the  actual  validity  of  the  cues. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  CUE 
INTEGRATION 
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23.  Title:  A  Theoretical  model  to  attack  the  enemy’s  decision-making  process 
Author(s):  Gibb,  R.  W. 

Source:  Paper  submitted  to  the  Naval  War  College,  Newport,  R.  I. 

Year:  2000 
Abstract: 

An  operational  commander  can  attack  enemy  vulnerabilities  because  of  human  susceptibilities  to 
biases  and  limitations  when  making  decisions.  This  paper  proposes  a  theoretical  model  to 
cognitively  attack  the  enemy  commander.  The  model  is  based  on  Information  Warfare  and 
Command  and  Control  Warfare,  as  well  as  psychological  human  information  processing  concepts. 
The  Observation-Orientation-Decision-Action  loop,  a  known  military  decision  cycle,  is  discussed 
and  contrasted  with  a  human  information  processing  model.  These  concepts  are  then  related  to  the 
planning  steps  of  the  Commander's  Estimate  of  the  Situation.  Human  decision-making  heuristics 
are  recommended  to  specifically  target  the  enemy's  Observation  and  Orientation  stages  of  the 
decision  cycle.  Consequently,  affecting  the  enemy's  operational  planning  of  mission  analysis  and 
courses  of  action.  The  model's  objective  is  to  alter  enemy  knowledge  and  beliefs,  thus  affecting 
situational  awareness  and  mental-model  development,  which  in  turn  will  produce  decisions  less 
favourable  for  the  enemy. 

Notes: 

Information  Warfare  (IW)  pertaining  to  information  psychology  consists  of  feeding  information  to 
the  enemy  to  have  them  act  as  you  desire.  The  goal  is  to  alter  their  thoughts  and  decision-making 
process  about  the  conflict.  The  ultimate  victory  would  be  to  convince  the  enemy  that  they  will  be 
unsuccessful  prior  to  the  start  of  armed  conflict.  The  author  relates  psychological  and  physiological 
process  models  to  decision-making  arguing  that  the  same  concepts  of  stress  and  task  saturation  are 
experienced  by  commanders  when  establishing  informational  cues.  Understanding  the  adversary’s 
culture  and  beliefs  is  necessary  to  understand  their  values  and  actions.  Biases  and  heuristics  can  be 
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utilized  to  further  deceive  the  enemy  into  establishing  false  situation  awareness  and  making  false 
conclusions  within  the  Observation-Orientation-Decision-Action  (OODA)  cycle.  The  author 
discusses  how  the  confirmation  bias,  availability  bias,  and  as-if  heuristics  can  be  exploited  in  IW. 
The  author  proposes  a  model  that  incorporates  the  OODA  loop  with  Commander’s  Estimate  of  the 
Situation  (CES)  and  information  processing,  emphasizing  that  attacking  stages  of  the  OODA  loop 
will  thus  indirectly  affect  later  stages  of  multiple  decision-making  processes  by  altering  beliefs  and 
mental  model  development. 

The  focus  if  this  article  is  on  military  decision  implies  a  DDM  context.  Specific  cognitive 
challenges  may  be  identified  through  looking  at  information  processing  such  as  is  highlighted  in 
this  article  (Wickens,  C.D.,  Flach,  J.M,  “Information  processing”  in  aviation  psychology,  1998  (19- 
41). 

The  author  relates  the  OODA  model  an  Observation-Orientation-Decision-Action  loop  to  the 
Wicken’s  and  Flach’s  model : 

Observation  (gathering  info  e.g.  reconnaissance)  =  short-term  sensory  store 

Orientation  (observing  info  and  converting  it  into  “intelligence”.  Creating  an  image  of  “reality  ”)= 
pattern  recognition 

Decision  (make  decision  on  “reality”  &  communicate  to  others)  =decision  and  response  selection 
Action  (execute  decision  actions)  =  response  execution 

Details  of  each  stage  are  only  briefly  described,  investigation  of  this  model  more  in-depth  may  be 
useful  in  understanding  pattern  recognition. 

The  OODA  model  was  used  to  demonstrates  some  cognitive  challenges  in  decision-making.  They 
include: 

identifying  relevant  information  within  the  environment  is  challenging  when  there  are 
multiple  sources  of  information  and  a  high  volume  of  information 

filtering  high  volumes  of  information  presents  problems  because  working  memory  has  a 
limited  processing  capacity.  (7  +/-  2) 

biases  influence  what  information  we  deem  is  relevant  to  the  decision  at  hand.  This  can 
then  effect  the  quality  of  the  decision  as  it  may  be  based  on  wrongly  “weighted”  values 
that  have  been  attributed  to  each  cue. 

People  have  difficulties  forecasting  in  partially  ambiguous  situations.  Predictions  is  often 
inaccurate. 

Attention  may  be  focused  on  particular  area  -  this  may  be  similar  to  biases.  Good 
explanation  of  what  is  meant  is  not  detailed. 

The  authors  present  a  theoretical  planning  model  (see  figure  below)  which  may  be  a  good  place  to 
start  in  identifying  where  DDM  in  operations  can  be  vulnerable;  analysis  of  the  mission,  factors 
affecting  possible  COAs,  development  of  COA,  analysis  of  COA  (own  and  enemy),  comparison  of 
COA,  may  all  be  influenced  by  biases  (specifically  identified  are  confirmation  bias,  availability 
bias,  “as-if’  heuristic). 
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Figure  2.  Theoretical  planning  model. 

Figure  5:  Planning  Model 


The  authors  also  identity  good  decision  -making  characteristics  they  include: 

1)  situational  awareness 

2)  meta-cognition 

3)  shared  mental  model 

4)  resource  management. 

These  may  be  areas  to  investigate  further  for  training  in  DDM. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  BIASES,  HEURISTICS  ,  general  challenges 
in  DDM 

To  be  used?  INCUDE  IF  RELEVANT 
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24.  Title:  Why  heuristics  work 
Author(s):  Gigerenzer,  G. 

Source:  Perspectives  on  Psychological  Science,  3,  20-29. 

Year:  2008 
Abstract: 

The  adaptive  toolbox  is  a  Darwinian-inspired  theory  that  conceives  of  the  mind  as  a  modular 
system  that  is  composed  of  heuristics,  their  building  blocks,  and  evolved  capacities.  The  study  of 
the  adaptive  toolbox  is  descriptive  and  analyzes  the  selection  and  structure  of  heuristics  in  social 
and  physical  environments.  The  study  of  ecological  rationality  is  prescriptive  and  identifies  the 
structure  of  environments  in  which  specific  heuristics  either  succeed  or  fail.  Results  have  been  used 
for  designing  heuristics  and  environments  to  improve  professional  decision  making  in  the  real 
world. 

Notes: 

The  author  discusses  heuristics  as  an  adaptive  way  of  making  decisions  in  complex  environments 
with  limited  information.  One  thing  stressed  is  that  heuristics  are  used  to  make  good  decisions;  in 
fact,  sometimes  the  decisions  are  better  than  “optimal”,  analytic  strategies. 

Several  relevant  and  important  points  made  relate  to  misunderstandings  of  heuristics  and  the  roles 
they  play,  including  (from  Table  1,  p.,  21): 

•  In  many  situations,  optimization  is  computationally  intractable  or  less  accurate  because  of 
estimation  errors  (this  means  that  heuristics  are  not  always  second-best  to  computational 
optimization,  and  are  sometimes  more  robust) 

•  Characteristics  of  the  environment  can  result  in  a  reliance  on  heuristics  (this  means  that  the 
use  of  heuristics  is  not  just  the  result  of  cognitive  limitations) 

•  Cognitive  capacity  appears  to  be  more  related  to  the  selection  of  appropriate  heuristics 
rather  than  reliance  on  computational  solutions  in  more  complex  situations  (i.e.,  reliance  on 
heuristics  is  not  more  prevalent  in  people  with  lower  cognitive  capability) 

•  Good  decisions  in  an  uncertain  world  often  require  ignoring  part  of  the  information 
(therefore  this  is  another  reason  why  analytic  solutions  are  not  always  better  than 
heuristics,  and  more  information  isn’t  always  better) 

In  this  article,  as  well  as  other  forecasting  articles,  the  author  points  out  the  need  to  predict  the 
future  and  not  just  understand  the  past.  The  author  indicates  that  cognitive  limitations  such  as 
forgetting  can  actually  add  to  the  robustness  of  decision  making,  possibly  by  reducing  noise  in  the 
system. 

The  author  points  out  that,  just  as  analytical  methods  are  not  always  better,  heuristics  are  not 
always  better;  one  has  to  assess  the  “ecological  rationality”  of  the  decision-making  strategy. 
Relevant  environmental  features  are  ones  which  predict  the  performance  of  the  problem  domain; 
some  possible  examples  are  the  size  of  learning  samples  and  the  amount  of  uncertainty  in  the 
domain. 

Table  2  (p.  24)  provides  a  list  of  possible  heuristics  that  may  be  contained  in  the  “adaptive 
toolbox”,  their  definitions,  conditions  for  ecological  rationality,  and  predictions. 

This  article  is  an  effective  argument  for  the  general  usefulness  and  adequacy  of  heuristics,  and 
seems  to  be  quite  useful  for  this  project  if  heuristics  is  the  topic  selected  for  elaboration.  It  does 
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deal  with  examples  from  DDM  environments  (e.g.,  making  investments)  and  addresses  cognitive 
challenges  (from  the  perspective  that  they  are  often  strengths  of  heuristics)  and  could  likely  be 
expanded  to  the  topic  of  training  (e.g.,  training  should  focus  on  understanding  environmentally 
rational  strategies,  rather  than  one  particular  strategy  to  be  generally  applied). 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS 
To  be  used?  INCLUDE  IF  RELEVANT 


25.  Title:  Challenges  supporting  cognitive  activities  in  dynamic  work  environments:  Application  to 
policing. 

Author(s):  Glantz,  E.  G. 

Source:  Dissertation  Abstracts  International  Section  A:  Humanities  and  Social  Sciences,  70  (1-A). 
Year:  2009 _ 

Abstract: 

Municipal  police  officers  perform  cognitive  activities,  such  as  decision  making,  judgment  and 
problem  solving,  while  facing  real  world  constraints  that  include  large,  socially  distributed  problem 
spaces  with  potential  for  high  risk  outcomes.  As  such,  the  goal  of  this  dissertation  is  to  improve 
cognitive  effectiveness  in  the  policing  domain  by  integrating  Cognitive  Systems  Engineering 
methodologies  to  achieve  three  objectives: 

1.  Developing  an  understanding  of  the  police  domain  and  its  constraints  on  the  cognitive  activities 
of  officers, 

2.  Analyzing  sources  of  complexity  and  needs  in  the  cognitive  activities  of  officers,  and 

3.  Exploring  technology  interventions  to  reduce  officer  task  complexity  and  accommodate  domain 
constraints. 

The  results  include  eight  findings  with  implications  for  cognitive  system  design.  The  findings 
suggest  it  is  possible  to  improve  the  use  of  technology  to  enhance  cognitive  effectiveness,  and  that 
there  are  strengths  in  combining  methodologies  to  explore  the  feasibility  of  cognitive  interventions. 

Notes: 

The  goals  of  this  dissertation  were  to  1 )  develop  an  understanding  of  the  police  domain  and  the 
effects  domain  constrains  have  on  the  cognitive  activities  of  officers;  2)  analyze  cognitive  tasks  and 
identify  sources  of  complexity  and  specific  needs  of  officers;  and  3)  explore  technological 
interventions  to  reduce  task  complexity  and  accommodate  domain  constraints. 

This  dissertation  describes  3  types  of  analyses  and  elements  within  them.  Decision  ladders,  concept 
maps,  Methodologies  include  interviews  as  forms  of  knowledge  elicitation  with  experts  (e.g.,  25 
years  of  experience),  attending  training  courses.  The  report  examined  the  complex  socio-technical 
systems  in  which  officers  make  dynamic  decisions.  This  is  particularly  influenced  by  political  and 
social  systems  in  which  they  operate. 

The  focus  of  the  data  analysis  was  the  development  and  evaluation  of  a  prototype.  The  purpose  was 
to  provide  understanding  into  cognitive  systems  and  may  lead  to  interventions  to  support  cognitive 
activities  in  the  field.  The  prototype  examined  was  a  scenario  training  that  provides  analytical 
situation  monitoring,  planning  and  control  decision  opportunities.  The  outcome  of  the  dissertation 
was  an  evaluation  of  the  effectiveness  of  designed  prototype  scenario  training  in  a  real-world _ 
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setting. 

The  author  explores  aspects  of  the  domain,  tasks,  and  decisions  officers  make  while  on  duty.  The 
topic  of  complex  decision  making  is  touched  upon,  though  is  not  the  focus  of  the  dissertation. 

There  is  discussion  of  the  relevant  elements  that  make  a  decision  complex,  and  workarounds  and 
potential  improvements  to  these  issues.  The  author  touched  on  aspects  of  the  system  in  which 
officers’  function  that  share  decision-making.  This  is  mainly  focused  on  the  technological  systems 
within  their  jobs. 

The  author  suggests  that  decision  complexities  are  due  to  the  elements  within  the  nature  of  the 
dynamic  environment  such  as:  accessing  information,  quality  (timeliness,  completeness,  accuracy) 
of  information,  if  information  must  be  shared  and/or  processed  by  others,  and  risks  to  the  worker 
and  the  environment.  Other  general  challenges  within  DDM  include  that  expectations  of  what  will 
occur  are  not  fully  developed  in  new  incidences  (mainly  due  to  lack  of  experience  and  lack  of 
information  about  the  situation).  Training  thus  may  focus  on  providing  more  DDM  experience,  and 
identifying  and  searching  for  more  (and  higher  quality)  information. 

The  cognitive  challenges  are  very  specific  to  the  situations  and  tasks  of  the  officers,  and  mainly 
focused  on  the  technological  systems,  not  DDM  per  se.  Discussion  pertaining  to  biases,  heuristics, 
cue  integration  and  pattern  recognition  are  not  explicit.  The  majority  of  this  dissertation  is  focused 
on  how  and  what  tasks  officers  performed,  rather  than  the  issues  surrounding  complex  decision 
making. 

The  authors  present  findings,  cognitive  design  implications  and  research  implications.  From  these  a 
number  of  cognitive  challenges  extracted  that  are  relevant  to  DDM  include: 

cognitive  demands  increase  as  the  scope  of  job  responsibilities  increases 
there  can  be  challenges  of  information  sharing  such  as  coordination  and  monitoring 
interoperability  reduces  effectiveness  and  results  in  unexplored  possibilities 
General  challenges  in  dynamic  environments  identified  include: 
accessing  information 

quality  of  information  (timeliness,  completeness,  accuracy) 
sharing  information  with  others 

relying  on  others  to  process  particular  information.  This  may  affect  the  quality  risks  to  the 
workers  and  the  environment 

CAN  BE  USED  TO:  Possible  cognitive  challenges  in  DDM. 

To  be  used?  FOR  CONSIDERATION 


26.  Title:  Does  intuition  beat  fast  and  frugal  heuristics:  A  systemic  empirical  analysis 
Author(s):  Glockner,  A. 

Source:  In  H.  Plessner,  C.  Betsch,  and  T.  Betsch  (Eds.),  Intuition  in  judgment  and  decision 
making  (pp.  309-325).  Mahwah,  NJ:  Lawrence  Erlbaum. 

Year:  2007 
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Abstract: 

(From  introduction): 

In  numerous  recent  publications  (Betsch,  2005;  Damasio,  1994;  Dougherty,  Gettys,  &  Ogden, 

1999;  Haidt,  2001 ;  kahneman  &  Frederick,  2002)  as  well  as  in  the  previous  chapters  of  this  volume 
(e.g.,  T.  Betsch,  chap.  1,  this  volume;  Deutsch  &  strack,  chap.  3,  this  volume;  epstein,  chap.  2,  this 
volume;  Hogarth,  chap.  6,  this  volume;  Weber  &  lindemann,  chap.  12,  this  volume),  the 
importance  of  automatic  processes  and  intuitive  strategies  in  decision  making  has  been  highlighted. 
Despite  the  increased  interest  in  this  topic,  rigorous  empirical  tests  that  measure  how  many  people 
indeed  use  such  intuitive  strategies  compared  with  simplifying  fast  and  frugal  heuristics 
(Gigerenzer,  Todd,  &  the  ABc  research  Group,  1999)  and  complex-rational  serial  weighted 
additive  strategies  (Payne,  Bettman,  &  Johnson,  1988)  are  still  rare.  At  least  two  reasons  for  this 
can  be  identified.  First,  some  of  the  automatic  models  make  it  hard  to  derive  testable  and  unique 
predictions  (e.g.,  Damasio,  1994;  slovic,  Finucane,  Peters,  &  McGregor,  2002).  second,  classical 
methods  to  detect  decision  strategies  like  information  search  analysis  (Payne  et  al.,  1988)  or 
analysis  of  think-aloud  protocols  (Montgomery  &  svenson,  1 983)  are  not  capable  of  detecting 
intuitive  decision  strategies  and  might  even  hamper  their  application  (Gldckner  &  Betsch,  2006; 
Hamm,  chap.  4,  this  volume).  My  aims  in  this  chapter  are  to  outline  a  method  that  avoids  these 
problems  and  to  present  data  from  a  research  program  that  closes  the  empirical  gap.  First,  I 
describe  the  relevant  classes  of  strategies — namely  complex-rational,  simplifying,  and  intuitive 
strategies — that  decision  makers  might  apply.  With  the  consistency-maximizing  strategy  (cMs; 
GlOckner,  2006),  I  introduce  one  specific  intuitive  decision  strategy  that  allows  for  deriving 
testable  and  distinguishable  predictions.  After  that,  I  outline  a  methodological  approach  that  allows 
for  detecting  the  intuitive  cMs.  Finally,  I  summarize  and  discuss  empirical  data  from  six 
experiments  that  shed  light  on  the  question  of  which  class  of  decision  strategies  accounts  for  the 
majority  of  individuals’  decision  behavior. 

Notes: 

The  author  proposes  an  intuitive  decision-making  strategy  and  a  methodological  approach.  Six 
empirical  experiments  are  discussed  that  address  the  question  of  which  class  of  decision  strategies 
accounts  for  the  majority  of  individuals’  decision  behaviour.  The  city  size  paradigm  tasks 
participants  with  deciding  which  of  two  cities  has  a  higher  population.  Dichotomous  cues  were 
used  in  the  majority  of  the  studies  reported.  Predictors  of  decision  strategies  are  cues  that  differ  in 
validity  (the  degree  to  which  they  are  correlated  to  the  criterion). 

The  author  argues  that  one-reason  decision  strategies  (take  the  best  heuristic,  TTB,  and  recognition 
heuristic,  REC)  and  tallying  strategies  like  equal  weight  rule  (EQW  or  Dawe’s  rule)  are  the  most 
important.  However,  recent  studies  show  conflicting  evidence  that  the  fast  and  frugal  heuristics 
proposed  by  Gigerenzer  et  al  (1999)  are  not  always  used  to  adapt  in  natural  environments  as 
proposed.  Choice  patterns  are  often  congruent  with  the  Weighted  Additive  Strategies  (WADD),  but 
sometimes,  it  is  argued,  are  best  explained  by  intuitive  decision  strategies.  One  process  that  has 
gained  support  as  being  that  which  individuals  use  in  decision  making  is  the  Parallel  Constraint 
Satisfaction  (PCS)  network  model.  In  this  model,  information  structures  are  set  in  to  automatic 
processes  when  individuals  perceive  a  decision  situation.  Information  alternatives  are  weighted 
against  each  other.  Information  is  modified  and  compared/matched  with  mental  representations  that 
might  be  used  in  decision  making.The  author  proposes  an  intuitive  Consistency-Maximizing 
Strategy  (CMS)  computational  algorithm  in  which  different  nodes  within  an  informational  structure 
are  excited  or  inhibited  and  then  excite  or  inhibit  others  leading  to  and  from  different  cues  and 
options.  This  framework  serves  to  form  a  representation  that  can  be  used  in  the  decision.  Pieces  of 
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information  are  weighted  against  each  other  and  the  weighting  can  be  modified  based  on  other  cues 
and  information  nodes  being  activated.  The  full  time  required  to  complete  this  process  and  come  up 
with  a  decision  is  the  “decision  time”. 

Methods  of  classification  are  discussed  and  limitations  are  pointed  out  (i.e.,  Bayesian  strategy 
which  does  not  allow  differentiation  between  the  WADD  and  CMS  methods  because  the  predicted 
choices  are  basically  equal).  The  author  then  proposes  a  better  method  of  classification  based  on 
the  analysis  of  choices  and  decision  times.  A  study  proposes  that  a  12-second  limitation  is  required 
to  apply  a  WADD’s  strategy  and  decision  times  below  his  method  would  reflect  random  strategy 
users.  The  second  step  would  then  classify  these  random  users  as  TTB,  EQW  or  CMS  using  the 
Bayesian  strategy  classification  method.  The  third  step  is  to  compare  decision  times  for  different 
decision  tasks  that  are  proposed  to  verify  classification  of  TTB  and  EQW. 

Memory-based  DM  may  be  different  than  other  DM.  Fast  and  frugal  heuristics  may  provide  the 
best  fit  for  this  type  of  DM. 

A  study  proposes  the  idea  that  with  different  time  limits  (severe  and  weak)  different  strategies  were 
used  (TTB  and  CMS  respectively ).Time  dependencies  for  the  use  of  different  strategies  could 
mean  that  difference  in  cue  dispersion  (using  the  computational  algorithm  idea)  may  lead  to  the 
application  of  different  heuristic  strategies. 

The  author  discusses  how  to  identify  different  decision  strategies.  Limitations  of  the  classifications 
methods  are  discussed.  Actual  content  of  decision  strategies  are  only  briefly  explained. 

This  article  may  be  suitable  in  supporting  theories  of  when  different  heuristics  or  DM  processing 
methods  may  be  used.  He  discusses  that  fast  and  frugal  heuristics  may  be  used  in  memory-based 
decisions,  and  decision  times  (e.g.,  12  s  minimum  for  use  of  WADD)  may  give  support  to  theories 
that  certain  types  of  decisions  require  particular  types  of  processing. 

This  article  challenges  others  that  put  forth  the  predominant  use  of  fast  and  frugal  heuristic. 

All  decision  making  tasks  were  simple  (not  dynamic). 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  the  use  and  categorization  of  HEURISTICS 
To  be  used?  FOR  CONSIDERATION 


27.  Title:  Fast  and  frugal  forecasting 
Author(s):  Goldstein,  D.  G.  &  Gigerenzer,  G. 

Source:  International  Journal  of  Forecasting,  25,  760-772. 

Year:  2009 
Abstract: 

Simple  statistical  forecasting  rules,  which  are  usually  simplifications  of  classical  models,  have  been 
shown  to  make  better  predictions  than  more  complex  rules,  especially  when  the  future  values  of  a 
criterion  are  highly  uncertain.  In  this  article,  we  provide  evidence  that  some  of  the  fast  and  frugal 
heuristics  that  people  use  intuitively  are  able  to  make  forecasts  that  are  as  good  as  or  better  than 
those  of  knowledge-intensive  procedures.  We  draw  from  research  on  the  adaptive  toolbox  and 
ecological  rationality  to  demonstrate  the  power  of  using  intuitive  heuristics  for  forecasting  in 
various  domains  including  sport,  business,  and  crime. 

Notes: 
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The  authors  investigate  forecasting,  and  the  possibility  that  simpler  equations  might  forecast  better 
than  more  complex  ones.  They  contrast  “fitting”,  or  matching  the  current  data  available,  with 
“forecasting”,  or  predicting  the  same  information  in  the  future;  shown  is  the  fact  that  using  4 
variables  will  typically  fit  the  data  worse  than  using  25  factors,  but  when  forecasting,  the  25-factor 
model  actually  has  higher  error.  This  is  called  “overfitting”  the  data  (when  fitting  current  data  is 
better  than  forecasting).  The  authors  call  a  model  “robust”  when  it  retains  accuracy  when 
comparing  fitting  and  forecasting. 

The  authors  note  that  the  usefulness  of  a  heuristic  is  related  to  features  of  the  information 
environment,  such  as  low  predictability.  Information  environments  cause  heuristics  to  succeed  or 
fail.  Their  Table  1  lists  a  number  of  heuristics  (recognition  heuristic,  fluency  heuristic,  take-the- 
best,  tallying,  hiatus  heuristic,  persistence  of  best,  1/N  heuristic,  center-of-the-circle),  conditions 
favouring  their  performance,  and  predictions  and  results. 

Simple  heuristics  have  more  than  just  simplicity  as  an  advantage;  they  are  also  transparent,  which 
makes  learning  easier. 

This  article  discusses  the  “cognitive  toolbox”,  as  mentioned  in  other  articles  (particularly  by 
Gigerenzer).  This  is  a  good  way  for  understanding  heuristic  reasoning  and  how  and  when  it  is 
useful.  Although  the  articles  is  not  written  in  a  DDM  context,  it  deals  with  the  issue  of  forecasting 
(i.e.,  predicting  the  future),  which  is  one  of  the  most  difficult  things  about  operating  in  a  DDM 
context.  It  also  discusses  fast  and  frugal  reasoning  which  would  be  useful  in  DDM  environments. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION 
To  be  used?  INCLUDE  IF  RELEVANT 


28.  Title:  Instance-based  learning  in  dynamic  decision  making. 

Author(s):  Gonzales,  C.,  Lerch,  J.  F.,  &  Lebiere,  C. 

Source:  Cognitive  Science:  A  Multidisciplinary  Journal,  27,  591-635. 

Year:  2003 _ 

Abstract: 

This  paper  presents  a  learning  theory  pertinent  to  dynamic  decision  making  (DDM)  called  instance- 
based  learning  theory  (IBLT).  IBLT  proposes  five  learning  mechanisms  in  the  context  of  a 
decision-making  process:  instance-based  knowledge,  recognition-based  retrieval,  adaptive 
strategies,  necessity-based  choice,  and  feedback  updates.  IBLT  suggests  in  DDM  people  learn  with 
the  accumulation  and  refinement  of  instances,  containing  the  decision-making  situation,  action,  and 
utility  of  decisions.  As  decision  makers  interact  with  a  dynamic  task,  they  recognize  a  situation 
according  to  its  similarity  to  past  instances,  adapt  their  judgment  strategies  from  heuristic-based  to 
instance-based,  and  refine  the  accumulated  knowledge  according  to  feedback  on  the  result  of  their 
actions.  The  IBLT's  learning  mechanisms  have  been  implemented  in  an  ACT-R  cognitive  model. 
Through  a  series  of  experiments,  this  paper  shows  how  the  IBLTs  learning  mechanisms  closely 
approximate  the  relative  trend  magnitude  and  performance  of  human  data.  Although  the  cognitive 
model  is  bounded  within  the  context  of  a  dynamic  task,  the  IBLT  is  a  general  theory  of  decision 
making  applicable  to  other  dynamic  environments. 

Notes: 

The  goal  of  this  article  is  to  explore  different  learning  theories  which  are  relevant  to  DDM. _ 
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Important  aspects  of  learning  include  learning  the  knowledge  base  (declarative  knowledge), 
recognition  (ability  to  identify  familiar  versus  unfamiliar  stimuli),  strategies  (adaptive  production 
systems),  and  evaluation  functions  (ability  to  assess  alternatives).  The  authors  argue  that  these 
theories  are  relevant  for  DDM  even  though  they  may  not  have  been  studied  in  that  context. 

The  authors  note  the  concept  of  satisficing  as  a  strategy,  and  also  note  that  the  level  of  satisfaction 
considered  “enough”  will  depend  on  the  time  remaining  to  make  a  decision.  How  the  decision 
maker  determines  when  to  intervene  is  also  highly  dependent  on  learning.  Feedback  is  also 
important,  but  is  often  not  adequate  in  DDM. 

It  appears  from  this  article  that  the  link  between  heuristics,  pattern  recognition  and  cue  integration 
are  not  clear  in  the  literature  -  in  some  cases,  it  appears  as  though  heuristics  rely  on  pattern 
recognition  and  cue  integration  as  the  situation  must  be  recognized  as  relevant  for  a  particular 
heuristic;  in  other  discussions,  it  appears  as  though  heuristics  are  thought  of  as  a  strategy  if  no 
pattern  is  recognized. 

Simulation  experiments  are  compared  to  human  data  to  try  to  uncover  the  cognitive  processes 
underlying  performance  in  DDM.  Learning  processes  investigated  include  random  choice  of 
strategies,  learning  instances,  recognition-based  retrieval,  adaptive  strategies,  necessity  level,  and 
feedback  updates.  Inexperienced  decision  makers  appear  to  act  like  the  random  alternative 
selection  simulation.  Over  time,  expertise  develops  and  humans  use  task  cues  to  determine  relevant 
strategies  and  use  these  to  select  responses.  The  model  which  best  fits  the  learning  trend  of  humans 
is  one  which  starts  random,  then  incorporates  knowledge  about  time,  then  retrieves  similar 
instances  and  blends  them  to  produce  an  estimate  of  the  best  response,  and  the  incorporation  of  a 
criterion  which  will  limit  the  time  to  make  a  decision.  The  model  with  feedback  mechanisms 
performed  better  than  the  humans,  so  the  model  without  feedback  was  more  accurate  at 
representing  human  performance. 

This  article  mentions  many  aspects  of  learning  theories  which  are  relevant  for  cue  integration  and 
pattern  recognition  including  the  use  of  chunks  and  templates  to  represent  information.  The  authors 
also  link  heuristics  and  pattern  recognition  by  stating  that  heuristics  are  determined  by  the 
recognition  of  familiar  patterns  of  information.  The  authors  also  relate  cue  integration  to  expertise, 
saying  that  experts  search  for  information  in  the  environment  much  more  selectively  than  novices. 

This  article  contains  interesting  information  about  possible  patterns  of  learning  and  learning  types 
which  are  applicable  to  learning  in  DDM.  There  is  contradictory  information  provided  about  the 
relationship  between  our  four  topics  of  interest.  The  application  of  a  simulation  to  learning  in  DDM 
is  interesting.  The  focus  of  this  article  is  more  on  determining  what  people  do,  but  it  could  also  be 
used  to  infer  information  about  how  DDM  could  be  trained. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION, 
AND/OR  PATTERN  RECOGNITION _ 

To  be  used?  INCLUDE  IF  RELEVANT 


29.  Title:  The  effects  of  team  leader  feedback  on  situation  assessment  in  distributed  anti-air  warfare 
teams 

Author(s):  Gough,  M.  J. 

Source:  Dissertation  submitted  to  the  Naval  Postgraduate  School,  Monterey,  CA. 
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Year:  1992 _ 

Abstract: 

Situation  assessment  is  the  first  step  in  the  Command  and  Control  Process.  In  naval  tactical  teams, 
it  has  become  more  critical  even  as  it  has  become  more  difficult.  Part  of  the  Navy's  attempt  to 
address  this  issue  is  the  Tactical  Decision  Making  Under  Stress  (TADMUS)  program.  Under 
TADMUS,  the  Situation  Assessment  In  Naval  Teams  (SAINT)  experiment  was  run  at  NPS  in 
December,  1991 .  This  thesis  describes  the  SAINT  experiment  and  uses  data  collected  during  the 
experiment  to  study  the  effects  of  team  leader  feedback  on  situation  assessment  in  distributed  air 
defense  teams.  The  emphasis  of  study  is  on  performance,  (error  rate  and  pattern),  subjective 
workload,  and  communication  rates.  Findings  include:  feedback  of  the  leader's  current  assessment 
lowers  explicit  coordination;  feedback  does  not  affect  subjective  workload;  feedback  increases 
error  rates,  and  may  affect  error  patterns.  Evidence  of  feedback  causing  confirmatory  bias  was  also 
found,  but  more  research  in  this  area  is  recommended. 

Notes: 

The  author  discusses  the  problem  of  anti-air  warfare;  the  navy  is  now  often  expected  to  act  in 
littoral  environments  which  require  faster  response  times  and  make  for  more  critical  errors;  the 
navy  is  more  accustomed  to  blue-water  engagements. 

This  writeup  is  based  on  the  result  of  an  experiment  from  the  TADMUS  (Tactical  Decision  Making 
Under  Stress)  project  which  examined  decision  making  in  the  context  of  anti-air  warfare.  A 
participant  acting  as  a  Tactical  Action  Officer  (TAO)  had  to  integrate  information  provided  by 
sensor  operators  and  make  a  hostility  determination  about  a  contact.  Feedback  about  the  current 
assessment  of  the  TAO  was  given  to  subordinates  in  the  “feedback”  condition;  this  feedback  was 
produced  by  the  participant  playing  the  role  of  the  TAO  and  not  manipulated  in  the  experiment. 

An  interesting  finding  related  to  training  was  that  giving  feedback  increased  perceived  workload, 
instead  of  lowering  it  as  predicted  (although  this  difference  was  not  statistically  significant).  There 
was  also  no  clear  relationship  between  feedback  and  error  rate;  in  fact,  in  some  conditions  there 
was  an  increase  in  errors  with  feedback.  Feedback  appears  to  increase  confidence  so  that  people  are 
willing  to  make  faster  decisions  on  less  evidence,  meaning  that  error  rates  increase  even  as 
confidence  in  the  decisions  increase.  In  other  words,  feedback  appears  to  add  to  confirmatory  bias. 

This  article  investigates  DDM  under  stress,  and  provides  interesting  information  about  the  possible 
relationships  between  leader  feedback  and  the  effect  of  some  biases  such  as  the  confirmation  bias. 
The  experiment  is  also  an  example  of  cue  integration  and  pattern  recognition,  as  the  TAO  has  to 
integrate  information  from  several  sources  and  then  make  a  decision  about  whether  the  pattern  is 
more  consistent  with  hostile  or  not  hostile  contacts. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  CUE 
INTEGRATION,  AND/OR  PATTERN  RECOGNITION _ 

To  be  used?  INCLUDE  IF  RELEVANT 


30.  Title:  Computational  cognitive  modeling  of  adaptive  choice  behavior  in  a  dynamic  decision 
paradigm 

Author(s):  Gray,  W.  D. 

Source:  Report  prepared  for  the  Air  Force  Office  of  Scientific  Research,  AFOSR  Report  number 
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F49620-03-1-0143. _ 

Year:  2006 _ 

Abstract: 

In  this  project,  we  braided  together  three  strands  of  research  to  study  how  interface  design  and 
decision  strategies  interact  to  affect  adaptive  choice  behavior  in  a  dynamic  decision  paradigm.  In 
the  first  strand  are  computational  models  of  dynamic  decision-making  built  within  a  well- 
recognized  cognitive  architecture,  ACT-R.  Our  second  strand  is  composed  of  the  decades-plus 
research  efforts  that  show  that  people  adopt  decision-making  strategies  that  tradeoff  effectiveness 
for  efficiency.  Of  particular  importance  is  work  showing  that  the  way  in  which  information  is 
presented  and  how  decision  aids  are  constructed  may  unintentionally  influence  the  decision 
strategies  adopted.  The  third  strand  is  work  from  our  own  lab  showing  the  influence  of  hard  and 
soft  constraints  on  routine  interactive  behavior.  This  work  suggests  that  the  cognitive  system  works 
locally  to  optimize  non-deliberate  choices  but  that  the  sum  of  such  locally  optimized  choices  may 
be  far  from  an  optimized  global  outcome.  These  strands  are  braided  together  in  Argus  Prime:  a 
simulated  task  environment  of  a  radar  operator’s  task  that  was  designed  to  study  the  interaction 
between  the  sometimes  conflicting  top-down  and  bottom-up  demands. 

Notes: 

This  is  a  review  of  a  program  of  research  being  carried  out  -  there  is  not  much  information  here 
beyond  a  very  brief  description  of  the  goals  of  research  and  some  approaches.  There  are  a  few 
references  given  that  may  prove  useful. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  Nothing _ 

To  be  used?  DO  NOT  INCLUDE  -  but  examine  for  references  that  may  prove  useful 


31.  Title:  Judgement  and  Decision  making  in  dynamic  tasks 
Author(s):  Hammond,  K. 

Source:  Report  prepared  for  the  United  States  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  ARI  Report  No.  88-81. 

Year:  1988 _ 

Abstract: 

A  theory  of  task  conditions  is  presented  on  the  ground  that  such  a  theory  is  a  prerequisite  for 
studying  dynamic  decision  making.  The  principal  features  of  the  theory  are  (a)  a  task-cognition 
inducement  principal,  (b)  a  distinction  drawn  between  surface  and  depth  characteristics  of  tasks, 
and  (c)  a  task  continuum  index.  Also  presented  is  a  theory  of  cognition  in  dynamic  tasks,  the  main 
features  of  which  are  (a)  a  cognitive  continuum  index  set  in  parallel  with  the  task  continuum  index, 
and  (b)  a  description  of  the  role  of  pattern  seeking  and  functional-relation  seeking  in  dynamic 
tasks.  The  practical  consequences  for  both  designers  and  operators  are  indicated. 

Notes: 

The  author  describes  all  features  within  a  theory  of  task  systems  and  a  theory  of  cognition  and 
relates  them  to  dynamic  tasks  and  cognition.  As  elements  within  the  task  change  cognitive 
conditions  change.  One  change  may  be  the  cognitive  approach  to  problem  solving  between 
rational,  intuition,  or  the  combination. 
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Pattern  seeking  is  induced  by  displays  of  info  that  provide  a  high  degree  of  perceptual  organization 
(visual,  auditory,  etc)  or  that  provide  high  degrees  of  conceptual  organization  (time  dependent 
events),  or  circumstances  that  require  the  subject  to  produce  a  coherent  explanation  of  his/her 
judgement  that  events  will,  have,  or  are  occurring. 

Seeking  functional  relations  are  induced  with  opposite  conditions  (e.g.,  not  perceptually, 
conceptually  organized,  not  demanding  immediate  coherent  explanations),  that  require  description 
and  prediction  of  amounts  of  variables. 

The  author  argues  there  are  different  types  of  templates  used  in  pattern  seeking  and  functional- 
relation  seeking  that  help  to  minimize  the  cognitive  workload  when  making  a  decision  about 
multiple  information  cues. 

Concludes  that  the  theoretical  framework  shown  in  the  paper  argues  that  the  explanation  of 
cognition  in  dynamic  tasks  should  begin  with  a  detailed  examination  of  task  systems. 

Changes  in  cognition  induced  by  changes  in  the  properties  of  task  systems  -  provides  some  insight 
as  to  how  the  nature  of  the  task  affects  cognition.  This  presents  the  idea  that  as  tasks  change,  how 
people  cognitively  process  decisions  within  that  task  may  change.  Thus  DDM  in  different  types  of 
tasks  may  use  different  approaches.  We  may  need  to  clarify  this,  which  tasks  we  want  to  examine 
further  to  understand  more  about  DDM  processes. 

The  article  may  be  useful  in  exploring  2  types  of  processes  suggested  to  be  required  in  cue 
integration  and  how  information  is  processed  as  required  for  DDM. 

A  lot  of  discussion  that  is  not  related  to  DDM  or  cognitive  challenges  etc.  is  in  this  article. 

Provides  a  good  theory  as  to  how  one  should  start  to  examine  the  cognition  as  it  relates  to  DDM. 
CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  General  cognition,  pattern  recognition. 

To  be  used?  INCLUDE  IF  RELEVANT 


32.  Title:  Exploring  the  entrepreneurial  mindset:  Feedback  and  adaptive  decision-making 
Author(s):  Haynie,  J.  M.  &  Shepherd,  D.  A. 

Source:  Manuscript  prepared  at  Syracuse  University 

Year:  2007 

Abstract: 

The  ability  to  sense  and  adapt  to  uncertainty  may  characterize  a  critical  entrepreneurial  and 
organizational  resource.  We  investigate  the  roles  that  metacognition  and  feedback-type  play  in 
facilitating  cognitive  adaptability:  the  ability  to  inform  and  adapt  a  previously  learned  decision 
heuristic  given  a  dynamic  task  environment.  Across  four,  inter-related  conjoint  experiments  we 
capture  and  decompose  more  than  10,000  “entrepreneurial  decisions”  nested  within  217 
individuals,  and  test  hypotheses  positioned  to  assess  the  concomitant  roles  of  metacognition  and 
feedback-type  in  the  decision-making  process.  Our  findings  suggest  that  metacognitive  awareness 
and  cognitive-based  feedback  are  positively  related  to  effective  adaptation  given  a  dynamic  task 
environment,  and  also  that  metacognitively  aware  individuals  use  cognitive-type  feedback  more 
effectively  than  individuals  less  metacognitively  aware. 

Notes: 
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Authors  describe  an  experiment  to  determine  the  effects  of  cognitive  (i.e.,  instruction  about  the 
underlying  relationships  between  variables)  versus  performance  (i.e.,  feedback  about  overall 
decision  quality  only)  feedback  on  performance  in  a  dynamic  environment.  They  hypothesized  that 
cognitive  feedback  would  be  more  effective  in  performance  than  performance  feedback;  as  well, 
metacognitive  awareness  would  moderate  the  relationship,  with  participants  with  more 
metacognitive  awareness  benefitting  more  from  cognitive  feedback  than  participants  with  less 
metacognitive  awareness.  Both  hypotheses  were  supported. 

Note  that  although  the  authors  talk  about  decision  making  in  a  dynamic  environment,  it  does  not 
appear  as  though  their  experience  actually  creates  one  -  especially  since  their  own  decisions  do  not 
appear  to  affect  the  decisions  they  have  to  make.  However,  it  is  likely  that  the  findings  are 
generalizable  to  DDM. 

The  article  does  focus  on  learning;  specifically,  what  kinds  of  feedback  (cognitive  versus 
performance)  impact  the  quality  of  decision  making  and  how  feedback  interacts  with  metacognitive 
awareness  to  affect  decision  making. 

Interestingly,  the  authors  mention  evidence  that  metacognitive  ability  can  be  taught.  This  indicates 
that  it  should  be  trained,  particularly  in  DDM  environments  where  cognitive  feedback  is  available 
as  people  high  in  metacognition  make  better  use  of  feedback.  Using  metacognition  may  be  loosely 
considered  a  heuristic  during  decision  making.  As  well,  the  article  mentions  that  a  major  part  of 
decision  making  in  complex  environments  is  understanding  how  to  incorporate  information,  which 
is  similar  to  the  cue  integration  problem. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION  (with 
broad  definitions). 

To  be  used?  INCLUDE  IF  RELEVANT 


33.  Title:  Human  Factors  Influencing  Decision  Making 
Authors):  Jacobs,  P.  A.  &  Gaver,  D.  P. 

Source:  Naval  Postgraduate  School,  Monterey,  California.  NPS-OR-98-003 

Year:  1998 

Abstract: 

This  report  supplies  references  and  comments  on  literature  that  identifies  human  factors 
influencing  decision  making,  particularly  military  decision  making.  The  literature  has  been 
classified  as  follows  (the  classes  are  not  mutually  exclusive):  features  of  human  information 
processing;  decision  making  models  which  are  not  mathematical  models  but  rather  are  descriptive; 
non-  personality  factors  influencing  decision  making;  national  characteristics  influencing  decision 
making;  personality  factors  influencing  decision  making;  decision  making  in  a  military 
organization.  The  decision  maker  is  influenced  by  many  factors  both  internal  to  the  decision  maker 
and  external  to  him/her.  The  environmental  context  in  which  a  decision  is  made  makes  it  difficult 
to  associate  personality  traits  with  specific  decision  making  behavior.  Internal  factors  that  influence 
decision  making  include  limited  information  processing  and  memory  capabilities.  These  limitations 
can  result  in  biases  in  processing  information  such  as  anchoring  (undue  weight  for  evidence 
supporting  (the  initial  hypothesis)  or  recency  (undue  weight  on  more  recent  evidence).  The 
limitations  can  also  result  in  decision  making  heuristics.  Training  and  experience  can  lessen  the 
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effects  of  limited  information  processing  and  memoiy  capabilities.  The  first  part  of  the  report  is  a 
summary  of  the  findings  of  the  literature  survey.  This  is  followed  by  detailed  endnotes  concerning 
the  references. 

Notes: 

Essentially  a  listing  of  loosely  annotated  references,  this  report  draws  attention  to  many  pieces  of 
literature  about  the  decision  maker  and  factors  that  affect  decision  making. 

Five  specific  articles  were  noted  as  being  relevant  in  the  current  project: 

Payne,  J.W.,  Bettman,  J.R.,  and  Johnson,  E.J.  (1992).  Behavioural  decision  research:  a 
constructive  processing  perspective.  Annual  Review  of  Psychology,  43,  pp  87-131.  About 
task  complexity:  “A  hypothesis  is  that  the  more  complex  a  decision  problem  the  more 
people  will  use  simplifying  heuristics”. 

Shanteau,  J.,  and  Stewart,  T.R.  (1992).  Why  study  expert  decision  making?  Some  historical 
perspectives  and  comments.  Organizational  Behaviour  and  Human  Decision  Processes,  52, 
pp  95-106. 

Einhom,  H.J.,  and  Hogarth,  R.M.  (1981).  Behavioural  decision  theoiy:  processes  of 
judgement  and  choice.  Annual  Review  of  Psychology,  32,  pp  53-88 

Davies,  D.R.  and  Parasuraman,  R.  (1982).  The  Psychology  of  Vigilance.  London: 
Academic:  “High  levels  of  perceived  stress,  negative  affect,  or  framing-induced  anticipated 
potential  losses  may  have  little  or  marginal  effect  on  simple  perceptual  tasks  performance 
but  appear  to  have  a  potentially  debilitating  effect  on  developing  and  using  more 
cognitively  complex  decision  making  strategies.  Individuals  experiencing  these  influences 
appear  to  do  more  than  simply  tunnel  their  focus  and  disproportionately  weight  negative 
information.  Framing  effects  may  do  more  than  merely  change  focus;  they  can  be  so 
disruptive  as  to  be  detrimental  to  the  incorporation  and  adoption  of  new  information  into 
dynamic  decision  making  strategies.” 

Cohen,  M.S.  and  Freeman,  J.T.  (1996).  Metarecognition  in  time-stressed  decision  making: 
recognizing,  critiquing  and  correcting.  Human  Factors,  38,  pp  206-219. 

These  articles  were  subsequently  reviewed  and  selected  articles  were  added  to  the  list  of  articles  to 
be  reviewed. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  NONE  OF  THE  SPECIFIED  AREAS. _ 

To  be  used?  DO  NOT  INCLUDE. 


34.  Title:  Cognitive  Biases  in  Military  Decision  Making 
Author(s):  Janser,  J. 

Source:  Paper  presented  to  the  U.S.  Army  War  College,  PA 

Year:  2007 _ 

Abstract: 

This  paper  examines  the  applicability  of  recent  findings  from  behavioral  economics  to  military 
decision  making.  Army  manuals  concerning  the  Military  Decision  Making  Process  mention  general 
biases  in  decision  making  but  do  not  mention  specific  biases  or  specific  mechanisms  for  mitigating 
bias.  Recent  research  has  shed  light  on  specific  biases  to  include:  overconfidence,  insensitivity  to 
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sample  size,  availability,  illusionary  correlation,  retrievability  of  instances,  escalation,  break  even, 
snake  bite,  fear  of  regret,  and  the  confirmation  bias.  The  Military  Decision  Making  Process  has  a 
long  and  distinguished  record  of  success.  However,  there  are  also  examples  of  military  failures  due 
to  cognitive  bias.  These  failures  include  Lee  at  Gettysburg  and  McClellan  in  Virginia.  Private 
industry  and  some  elements  of  the  Army  have  started  to  account  for  these  deficiencies  through 
various  practices  including  coaching  and  training.  This  paper  concludes  that  the  Military  Decision 
Making  Process  as  described  in  FM  5-0  is  deficient  in  not  fully  recognizing  and  accounting  for 
cognitive  biases.  The  process  can  be  improved  through  several  steps.  These  steps  include  not  only 
research,  education,  and  training,  but  also  procedural  and  organizational  changes. 

Notes: 

The  author  identifies  a  systematic  process  for  complex  decision  making,  Military  Decision  Making 
Process  (MDMP),  trained  in  the  US  army;  unfortunately  time  is  required  to  follow  through  each 
step,  which  is  not  commonly  available.  This  method  also  requires  particular  conditions:  an 
accurately  defined  mission,  accurately  assessed  enemy  and  friendly  situations,  all  alternatives 
identified,  all  criteria  identified,  all  criteria  accurately  weighted,  each  COA  (course  of  action) 
accurately  assessed,  and  a  correctly  made  decision  based  on  a  COA  comparison.  Biases  affect  all  of 
these  conditions.  This  paper  examines  a  number  of  different  biases  and  attempts  to  mitigate  biases 
in  military  decision  making  by  explicitly  identifying  the  different  types  and  threats  they  pose. 

The  conclusions  based  on  these  elements  include  that  weaknesses  of  human  decision  making  are 
exacerbated  at  the  most  critical  times  such  as  in  combat  operations  when  fatigue,  stress,  time 
limitations  and  incomplete  information  are  high.  Psychologist  coaching  and  training  has  worked  in 
the  industrial  sector  to  help  improve  performance.  Suggestions  are  provided  that  include 
incorporating  further  research,  education,  procedural  changes,  training,  and  organizational  changes 
that  may  improve  the  influence  of  biases  when  making  decisions  using  the  MDMP. 

The  author  provides  descriptions  of  biases  such  as  overconfidence,  insensitivity  to  sample  size, 
availability,  illusionary  correlation,  retrievability  of  instances,  escalation,  break  even,  snakebite, 
fear  of  regret,  and  the  confirmation  bias. 

The  author  believes  that  educating  decision  makers  on  the  nature  and  type  of  biases  would  improve 
military  decision  making  performance  significantly.  This  idea  refuted  in  more  recent  studies. 

This  report  is  useful  for  gaining  an  understanding  of  the  different  types  of  decision  making  biases 
though  little  on  their  application,  or  ideas  that  might  be  useful  for  training. 

The  decision  making  technique  is  helpful  in  understanding  the  approach  military  decision  makers 
may  use,  though  clearly  the  use  of  this  planned  strategy  is  an  ideal ,  and  expected  to  be  rarely  used 
in  the  constraints  and  limitations  of  DDM.  Training  focused  on  how  best  to  shorten  the  MDMP 
process  may  correspond  with  the  use  of  heuristics,  or  other  improvements  to  DDM. 

The  MDMP  include  7  parts: 

5)  receipt  of  mission 

6)  mission  analysis 

7)  course  of  action  (COA)  development 

8)  COA  analysis 

9)  COA  comparison 

10)  COA  approval 

1 1 )  Orders  production 


Human systems' 


Cue  integration  in  Dynamic  Decision  making 


Page  105 


HU  HAH  SYSTEMS 


CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  BIASES 
To  be  used?  FOR  CONSIDERATION 


35.  Title:  Patterns  of  Error:  Perceptual  and  Cognitive  Bias  in  Intelligence  Analysis  and  Decision- 
Making 

Author(s):  Jones,  L. 

Source:  Master’s  thesis  presented  at  the  Naval  Postgraduate  School,  Monterey,  California. 

Year:  2005 
Abstract: 

The  history  of  man  is  written  in  choice.  Whether  simple  or  complex,  on  a  whim  or  after  labored 
consideration,  inflamed  by  passion  or  calculated  coolly,  the  judgments  that  we  form  and  the 
choices  that  we  make  define  who  we  are  and  what  we  want  for  the  future.  Yet  most  of  us  have  little 
or  no  conscious  awareness  of  the  inner  workings  of  our  own  minds.  We  often  choose  without 
understanding  or  accounting  for  the  perceptions,  intuitions,  and  inferences  that  underlie  our 
decisions.  So  how  do  people  make  decisions?  How  do  we  cope  with  the  volume  and  complexity  of 
information  in  our  environment  without  being  overwhelmed?  How  do  we  use  our  senses  to  select 
and  process  this  information,  and  how  do  we  organize,  contextualize,  and  conceptualize  it  once  it 
reaches  our  brains?  How  do  we  form  judgments  about  the  value  of  a  specific  piece  of  information 
or  about  the  likelihood  of  a  particular  event  or  outcome?  And  what  are  the  factors  that  lead  us 
astray?  The  search  for  answers  to  these  questions  is  more  than  academic;  understanding  the 
fundamentals  of  perception  and  cognition  is  critical  to  effective  analysis  and  decision-making.  For 
those  involved  in  national  security,  and  particularly  for  those  involved  in  the  collection  and 
analysis  of  national  intelligence,  an  appreciation  of  the  intricacies  of  these  processes  has  real-world 
implications.  As  evidenced  by  the  dramatic  intelligence  failures  of  the  last  few  years,  and  in 
particular  by  the  mistaken  assessment  concerning  Iraqi  weapons  of  mass  destruction,  understanding 
how  we  arrive  at  judgments  and  decisions  can  be  quite  literally  a  matter  of  life  and  death. 

Notes: 

This  thesis  takes  a  brief  walk  from  the  classic  notions  of  rational  decision  making  into  bounded 
rationality  and  finally  links  to  the  development  of  theories  of  heuristics  and  biases.  Attention  is 
paid  to  the  limited  knowledge  and  computational  ability  of  decision  makers,  the  creation  of  mental 
models  and  the  existence  of  sophisticated  cognitive  processes.  Specific  aspects  of  perception  are 
discussed  (e.g.  perception  is  unavoidable  and  it  constructs  rather  that  records  reality;  perception  is 
influenced  by  expectations  and  situational  context;  exposure  to  blurred/ambiguous  stimuli 
interferes  with  perception  accuracy)  as  well  as  three  aspects  of  cognition  (a.  mental  models  are 
unavoidable,  b.  mental  models  are  quick  to  form  but  resistant  to  change,  c.  new  information  is 
assimilated  to  existing  content).  The  representative,  availability  and  anchoring  heuristics  are  then 
introduced.  Finally,  various  intelligence  conclusions  about  Iraq  later  proven  incorrect  are  discussed 
and  causes  of  these  errors  related  to  perception,  decision  making,  heuristics  and  biases  are 
postulated. 

The  author  provides  a  well  written  progress  of  decision  making  research  linking  together  the 
identification  of  specific  cognitive  limitations  to  the  development  of  new  theories  in  decision 
making.  There  is  no  specific  discussion  about  dynamic  decision  making. 
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CAN  BE  USED  TO  SUPPORT  INVESTIGATING  CUES,  PATTERN  RECOGNITION, 
HEURISTICS,  AND  BIASES  -  BUT  IS  NOT  DDM. 

To  be  used?  FOR  CONSIDERATION 


36.  Title:  The  potential  for  social  contextual  and  group  biases  in  team  decision-making:  Biases, 
conditions  and  psychological  mechanisms. 

Author(s):  Jones,  P.  E.  &  Roelofsma,  P.  H.  M.  P. 

Source:  Ergonomics,  43, 1 129-1152. 

Year:  2000 _ 

Abstract: 

This  paper  provides  a  critical  review  of  social  contextual  and  group  biases  that  are  relevant  to  team 
decision-making  in  command  and  control  situations.  This  paper  provides  an  insight  into  the 
potential  that  these  types  of  biases  have  to  affect  the  decision-making  of  such  teams.  The  biases 
considered  are  false  consensus,  groupthink,  group  polarization,  and  group  escalation  of 
commitment.  For  each  bias,  the  following  questions  are  addressed:  What  is  the  descriptive  nature 
of  the  bias?  What  factors  induce  the  bias?  What  psychological  mechanisms  underlie  the  bias?  What 
is  the  relevance  of  the  bias  to  command  and  control  teams?  The  analysis  suggests  that  these  biases 
have  a  strong  potential  to  affect  team  decisions.  Consistent  with  the  nature  of  team  decision¬ 
making  in  command  and  control  situations,  all  of  the  biases  tend  to  be  associated  with  those 
decisions  that  are  important  or  novel  and  are  promoted  by  time  pressure  and  high  levels  of 
uncertainty.  A  concept  unifying  these  biases  is  that  of  the  shared  mental  model,  but  whereas  false 
consensus  emanates  from  social  projection  tendencies,  the  rest  emanate  from  social  influence 
factors.  The  authors  also  discuss  the  “tricky”  distinction  between  teams  and  groups  and  propose  a 
revised  definition  for  command  and  control  team. 

Notes: 

Four  specific  group  biases  are  discussed  in  detail:  false  consensus  effect,  groupthink,  group 
polarization,  and  group  escalation  of  commitment.  These  biases  are  discussed  in  the  context  of 
command  and  control  teams.  For  each,  there  is  a  description  of  the  bias,  results  of  studies  of  the 
bias,  possible  underlying  psychological  mechanisms,  and  a  discussion  about  the  implications  of  the 
bias  specifically  for  command  and  control  teams. 

False  consensus  effect  is  the  “tendency  to  overestimate  the  degree  of  similarity  between  self  and 
others”  (p  1 134).  A  strong  and  well  documented  bias,  possible  causes  include  the  availability 
heuristic  and/or  a  desire  for  social  support.  Problems  with  information  estimation,  distortion  and 
assumptions  about  others  (e.g.  a  commander’s  intentions)  are  conceivable  issues  in  command  and 
control  groups. 

Groupthink  is  the  ability  for  poor  decisions  to  come  from  otherwise  capable  and  well  intentioned 
groups.  This  effect  tends  to  arise  when  groups  are  highly  driven  by  unanimity,  a  condition  relevant 
to  hierarchical  military  teams  with  little  to  no  outside  help. 

Group  polarization  is  the  intensification  of  a  position  on  an  issue  held  by  the  majority  and  group 
escalation  of  commitment  is  the  tendency  to  support  an  action  path  even  though  it  is  failing. 

Though  not  explicitly  discussed,  it  would  be  reasonable  to  assume  that  DDM  could  inherently  be  a 
part  of  command  and  control  decision  making.  Thus,  if  there  is  interest  in  grouji  biases  this  article 


Hum&nsystems' 


Cue  integration  in  Dynamic  Decision  making 


Page  107 


HUMANS  >’$r£jl/£ 


would  be  worthwhile. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  BIASES. 
To  be  used?  INCLUDE  IF  RELEVANT 


37.  Title:  Exemplar  effects  in  categorization  and  multiple-cue  judgment 
Author(s):  Juslin,  P.,  Olsson,  H.,  &  Olsson,  A.-C. 

Source:  Journal  of  Experimental  Psychology:  General,  132,  133-156 

Year:  2003 _ 

Abstract: 

Categorization  and  multiple-cue  judgment  are  similar  tasks,  but  the  influential  models  in  the  two 
areas  are  different  in  terms  of  the  computations,  processes,  and  neural  substrates  that  they  imply.  In 
categorization,  exemplar  memory  is  often  emphasized,  whereas  multiple-cue  judgment  generally  is 
interpreted  in  terms  of  integration  of  Cues  that  have  been  abstracted  in  training.  In  3  experiments 
the  authors  investigated  whether  these  conclusions  derive  from  genuine  differences  in  the  processes 
or  are  accidental  to  the  different  research  methods.  The  results  revealed  large  individual  differences 
and  a  shift  from  exemplar  memory  to  cue  abstraction  when  the  criterion  is  changed  from  a  binary 
to  a  continuous  variable,  especially  for  a  probabilistic  criterion.  People  appear  to  switch  between 
qualitatively  distinct  processes  in  the  2  tasks. 

Notes: 

Discusses  the  similarity  and  differences  between  two  tasks  or  research  paradigms;  categorization 
and  multiple-cue  judgment.  These  appear  to  basically  be  the  same  task,  because  participants  are 
asked  to  interpret  cues  and  make  inferences  based  on  those  cues.  The  main  differences  are  that  in 
categorization  the  inference  is  a  determination  of  which  of  a  small  number  of  categories  the 
stimulus  belongs  to  (usually  only  2  categories),  whereas  the  judgment  in  multiple-cue  judgment  is  a 
continuous  variable  (e.g.,  the  probability  of  precipitation  given  weather  information).  Because  the 
two  areas  of  research  have  arrived  at  very  different  conclusions  about  each  of  the  tasks,  the  authors 
have  decided  to  examine  whether  there  is  actually  evidence  that  these  are  different  processes,  or  the 
same  process  examined  in  different  ways. 

Exemplars  are  usually  incorporated  into  currently  successful  models  of  categorization.  Exemplars 
represent  specific  examples  provided  by  experience  which  are  then  applied  to  new  cases.  If  the 
exemplar  is  similar  enough  to  the  new  case,  then  inferences  are  made  about  the  new  case  based  on 
the  old  one.  Multiple-cue  judgments,  on  the  other  hand,  are  typically  represented  via  multiple- 
linear  regression  models,  which  assume  that  people  integrate  information  about  cues  according  to  a 
linear  additive  rule  (i.e.,  information  from  cues  are  added,  and  optimal  weights  are  also 
remembered  and  used  in  making  the  judgment).  These  theories  would  call  on  different  neural 
processes  and  make  different  predictions  based  on  the  role  of  memory. 

Based  on  three  experiments,  the  authors  conclude  that  differences  between  the  categorization  and 
multiple-cue  judgment  tasks  actually  produce  qualitatively  different  cognitive  processes.  Mental 
cue  abstraction  is  favoured  when  a  continuous  criterion  is  used,  whereas  a  binary  criterion  tends  to 
be  interpreted  via  exemplar  memory.  The  authors  indicate  that  this  might  be  due  to  more 
informative  feedback  in  the  continuous  judgment  paradigm.  The  authors  propose  that  if  feedback  is 
sufficient  to  infer  relations  between  cues  and  the  criterion  used  for  judgment,  then  multiple  cue 


Page  108 


Cue  integration  in  Dynamic  Decision  making 


Hnmamystems 


HUMAN  SYSTEMS 


abstraction  will  be  favoured.  With  insufficient  feedback  rules  cannot  be  inferred,  so  exemplar 
memory  is  the  most  appropriate  guide.  The  authors  did  also  notice  a  large  individual  difference  in 
the  preference  for  exemplar  memory  or  cue  abstraction. 

In  DDM,  there  are  likely  many  instances  of  each  type  of  decision;  as  well,  many  decisions  could 
probably  be  imagined  as  both.  For  example,  a  judgment  could  involve  “friend  or  foe?”  -  is 
someone  an  enemy  or  not,  which  would  be  a  categorization  judgment.  The  same  type  of  judgment 
could  be  seen  as  a  multiple-cue  judgment,  if  someone  were  trying  to  infer  the  probability  of  danger. 

From  the  results  of  this  experiment,  it  seems  as  though  the  level  of  feedback  in  different  important 
tasks  should  be  examined  to  determine  whether  cue  abstraction  is  a  viable  method  of  making 
decisions.  Although  cue  abstraction  might  be  a  preferred  method  of  making  decisions,  there  may 
not  be  adequate  information  for  it  to  be  used  effectively.  As  well,  as  there  are  individual 
differences  in  decision-making  preference  between  the  two  methods,  perhaps  people  should  be 
trained  in  their  natural  style,  or  they  may  otherwise  not  perform  optimally,  especially  under  stress 
and  time  pressure. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  CUE  INTEGRATION  AND  PATTERN 
RECOGNITION _ 

To  be  used?  INCLUDE  IF  RELEVANT 


38.  Title:  Determinants  of  linear  judgment:  A  meta-analysis  of  lens  model  studies 
Author(s):  Karelaia,  N.  &  Hogarth,  R.  M. 

Source:  Psychological  Bulletin,  134, 404-426 

Year:  2008 

Abstract: 

The  mathematical  representation  of  E.  Brunswik’s  (1952)  lens  model  has  been  used  extensively  to 
study  human  judgment  and  provides  a  unique  opportunity  to  conduct  a  meta-analysis  of  studies  that 
covers  roughly  5  decades.  Specifically,  the  authors  analyzed  statistics  of  the  “lens  model  equation” 
(L.  R.  Tucker,  1964)  associated  with  249  different  task  environments  obtained  from  86  articles.  On 
average,  fairly  high  levels  of  judgmental  achievement  were  found,  and  people  were  seen  to  be 
capable  of  achieving  similar  levels  of  cognitive  performance  in  noisy  and  predictable 
environments.  Further,  the  effects  of  task  characteristics  that  influence  judgment  (numbers  and 
types  of  cues,  inter-cue  redundancy,  function  forms  and  cue  weights  in  the  ecology,  laboratory 
versus  field  studies,  and  experience  with  the  task)  were  identified  and  estimated.  A  detailed 
analysis  of  learning  studies  revealed  that  the  most  effective  form  of  feedback  was  information 
about  the  task.  The  authors  also  analyzed  empirically  under  what  conditions  the  application  of 
bootstrapping —  or  replacing  judges  by  their  linear  models — is  advantageous.  Finally,  the  authors 
note  shortcomings  of  the  kinds  of  studies  conducted  to  date,  limitations  in  the  lens  model 
methodology,  and  possibilities  for  future  research. 

Notes: 

One  of  the  authors’  goals  is  to  assess  how  task  characteristics  influence  judgment,  using  the  lens 
model  as  a  decision-making  strategy.  Lens  model  studies  are  aggregated  and  a  meta-analysis 
performed  to  determine  factors  that  influence  decision-making  performance. 

The  lens  model,  in  brief,  is  a  model  of  judgement  which  states  that  human  judgment  is  a  linear 
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function  of  a  set  of  cues.  The  cues  are  weighted,  and  an  error  term  is  added  which  represents  an 
error  term.  The  environmental  criterion  (i.e.,  the  actual  value  of  what  the  judgment  is  about;  what 
the  human  judgment  is  aiming  to  estimate)  is  represented  the  same  way.  The  difference  between 
the  two  values  represents  the  error  in  judgment.  In  addition,  this  is  a  probabilistic  model,  as  the 
cues  are  never  perfectly  valid  or  reliable. 

The  authors  cite  evidence  that  environments  and  judges  are  generally  well  modeled  by  linear 
functions  (such  as  the  lens  model). 

There  is  a  discussion  of  bootstrapping;  this  means  replacing  a  person’s  judgment  with  their  strategy 
(i.e.,  calculating  their  linear  formula  and  then  applying  this  formula).  Some  studies  show  an 
advantage  of  bootstrapping  in  several  domains  including  clinical  decision  making  and  employment 
interviews.  Bootstrapping  is  better  when  the  judge  has  valid  linear  knowledge  and  the  environment 
is  predictable. 

In  their  meta-analysis  the  authors  examined  different  factors  which  influence  judgment 
performance.  Their  findings  include: 

•  Number  of  cues:  Judgment  accuracy  was  highest  with  two  cues,  and  as  the  number  of  cues 
increased  accuracy  decreased 

•  Cue  redundancy:  Judgment  accuracy  decreased  with  cue  redundancy 

•  Function  form:  Judgment  accuracy  was  higher  with  linear  functions  between  cues  than 
nonlinear  functions 

•  Cue  weights:  Judgment  accuracy  was  higher  with  equally  weighted  cues 

•  Laboratory  vs.  field  studies:  When  controlling  for  other  factors  (e.g.,  number  of  cues), 
accuracy  was  equivalent 

•  Abstract  vs.  concrete:  Performance  was  better  with  concrete  compared  to  abstract  tasks 

•  Expertise:  expertise  was  negatively  related  to  judgement  accuracy,  but  positively  related  to 
judgmental  consistency;  however,  learning  did  occur  over  multiple  trials 

The  authors  offer  three  major  conclusions  (p.  420): 

•  People  are  capable  of  achieving  high  levels  of  judgmental  performance 

•  People  learn  best  from  feedback  that  instructs  them  about  the  characteristics  of  the  tasks 
they  face 

•  The  inconsistency  that  people  exhibit  in  making  judgments  is  sufficient  for  models  of  their 
judgments  to  be  more  accurate  than  they  are  themselves 

This  is  a  useful  article  which  explains  the  lens  model  and  provides  a  review  of  most  of  the  relevant 
literature  related  to  human  judgment.  While  not  explicitly  including  a  DDM  context,  the  findings 
should  be  extendable  to  that  domain  with  a  consideration  of  the  impact  of  DDM  characteristics 
(e.g.,  time  pressure,  constantly  changing  information).  A  discussion  of  cognitive  and  training 
challenges  should  address  the  bootstrapping  vs.  human  judgment  issue;  that  is,  when  should  a 
statistical  model  be  used  instead  of  a  human  decision  maker?  The  dichotomy  between  expertise 
hurting  performance  and  task  training  improving  performance  should  also  be  addressed. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  CUE  INTEGRATION 
To  be  used?  INCLUDE  IF  RELEVANT 
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39.  Title:  Sense  Making:  Biases  and  support  solutions 
Author(s):  Kerstholt,  J.  H. 

Source:  In  Tactical  Decision  Making  and  Situation  Awareness  for  Defence  Against  Terrorism  (3-1 
-  3-8).  Meeting  Proceedings  RTO-MP-SCI-174.  Neuilly-sur-Seine,  France:  RTO. 

Year:  2006 
Abstract: 

Analysts  base  their  hypotheses  concerning  terrorist  groups  and  terrorist  attacks  generally  on  large 
amounts  of  information,  which  may  be  uncertain,  inconsistent  and/or  incomplete.  The  present 
paper  addresses  the  questions  of  how  an  analyst  constructs  a  causal  scenario,  which  biases  occur 
and  how  this  intelligence  process  can  be  supported.  In  the  first  part  a  model  is  presented  that 
describes  the  subsequent  steps  in  the  analysis  process.  Several  biases  may  negatively  affect  the 
quality  of  the  analysis  process.  These  are  described  in  the  second  part  of  the  paper.  Subsequently, 
an  experiment  is  described  in  which  we  manipulated  the  presentation  of  information:  in  a  time  line, 
a  relation  scheme  or  unstructured.  In  general,  the  results  show  that  people  generate  fewer  motives 
in  the  unstructured  task  condition  and  that  expectations  increase  probability  assessment  of  related 
hypotheses.  The  last  part  of  the  paper  presents  support  concepts  which  aid  analysts  to  critically 
think  about  their  own  analysis:  To  what  extent  is  a  hypothesis  anchored  in  the  data  at  hand,  how  to 
explain  data  not  in  line  with  the  hypothesis  and  which  other  hypotheses  would  be  possible  as  well? 
Supporting  analysts  in  critical  thinking  not  only  improves  situation  awareness  and  decision  making, 
but  allows  for  better  communication  with  others  as  well. 

Notes: 

This  paper  looks  at  the  questions  related  to  how  analysts  construct  causal  scenario  on  which 
intelligence  is  based  in  the  context  of  terrorists  groups  and  terrorist  attacks.  Biases  that  occur  in  this 
process  are  discussed.  An  experiment  is  then  described  that  manipulates  the  presentation  of 
information  (time  line  format,  relation  scheme,  or  unstructured).  General  results  show  that  people 
generate  fewer  crime  motives  when  required  to  use  unstructured  data. 

The  author  describes  how  data  can  be  processed,  filtered,  and  used  to  build  a  comprehensive 
analysis  or  support  for  a  hypothesis  (depending  on  if  you  start  with  data  and  process  bottom-up  or 
start  with  a  story  and  process  top-down).  The  language  used  and  the  experience  of  the  analyst  can 
influence  their  interpretations  in  the  form  of  biases,  as  can  analysts’  expectations.  The  author 
concludes  that  awareness  alone  is  not  sufficient  to  eliminate  biased  interpretation. 

An  experiment  was  conducted  that  presented  information  (crime  scene  /scenario  data)  in  different 
forms  1)  in  a  timeline  fashion;  2)  as  data  related  to  relationships  of  the  victim;  and  3)  in  an 
unstructured  form.  Results  showed  fewer  motives  were  devised  by  participants  in  the  unstructured 
task  condition  compared  with  the  other  two.  Possible  explanations  include  that  there  is  an  increased 
workload  required  to  arrange  data  in  the  unstructured  condition.  The  author  proposes  that  meta- 
cognitive  strategies  enable  critical  evaluation  of  data.  Discussions  focus  on  the  analyst’s  ability  to 
make  judgements  and  the  biases  they  encounter. 

This  review  discusses  data  processing  and  potential  biases,  however  not  in  the  context  of  DDM. 

Some  information,  maybe  useful  in  transferring  to  DDM  such  as  the  fact  that  time  is  required  and 
cognitive  resources  are  required  to  make  appropriate  assessments  when  data  is  presented  in  an 
unstructured  way. 

A  model  of  the  intelligence  process  is  presented  (from  Pirollo  &  Card,  2005) 
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Figure  6:  Model  of  Intelligence  Process  (Pirollo  &  Card,  2005) 

Understanding  how  scheme  are  developed  may  demonstrate  the  challenges  within  the  development 
and  use  of  heuristics  and  biases.  The  confirmation  bias  is  the  main  point  of  discussion.  It  is  pointed 
out  that  when  workload  increases,  confirmation  bias  is  more  likely  to  occur.  Similarly,  the  way  the 
information  is  presented,  the  type  of  language  used,  and  our  own  past  experiences  can  greatly 
influence  our  interpretations  in  the  forms  of  biases. 

The  authors  state  the  main  threats  to  biased  information  processing  are  the  influence  of  prior 
beliefs,  perspectives,  and  their  mindset.  They  suggest  that  increasing  the  amount  of  information 
would  help  judges  make  a  better  assessment  of  the  situation.  This  however  is  not  agreed  upon  by 
all,  for  example  those  who  advocate  “less  is  more”  theory  (that  people  make  better  decisions  with 
less  information).  They  also  suggest  that  in-depth  analysis  of  critical  assumptions  underlying  their 
hypothesis  may  help  the  to  properly  evaluate  and  weight  information.  This  idea  is  not  relevant  to 
DDM  as  time  constraints  are  an  issue. 

An  important  counter  argument  is  to  note  that  they  conclude  simple  awareness  of  a  bias  is  not 
enough  to  eliminate  its  effects. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  BIASES _ 
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Author(s):  Kerstholt,  J.  H.  &  Willems,  P. 

Source:  TNO  Defence  Research,  Report  No.  IZF  1993  B-8 

Year:  1993 

Abstract: 

To  date,  research  has  mainly  used  deadlines  in  static  task  environments  to  investigate  the  effects  of 
time  pressure  on  decision  making  behaviour.  A  dynamic  environment,  on  the  other  hand,  changes 
over  time  and  time  pressure  may  result  when  time  is  not  used  efficiently  and  negative 
consequences  are  rapidly  increasing.  An  experiment  was  carried  out  to  investigate  the  effects  of 
time  horizon  on  decision  making  behaviour  in  a  dynamic  task  environment.  Subjects  were  required 
to  monitor  the  fitness  level  of  a  running  athlete,  depicted  graphically  on  a  computer  screen,  and  to 
apply  a  proper  treatment  whenever  necessary.  Information  could  be  requested  on  symptoms,  which 
were  probabilistically  related  to  underlying  causes.  The  time  horizon  was  manipulated  by  the  speed 
at  which  the  athlete's  fitness  level  changed  over  time.  Restrictions  in  time  horizon  did  not  affect  the 
amount  and  type  of  information  that  was  requested,  and  the  diagnoses  became  even  better. 
Nevertheless,  more  athletes  collapsed  in  the  more  restricted  time  horizon  conditions.  In  the  short 
time  horizon  conditions  subjects  furthermore  employed  a  more  cautious  decision  strategy.  As  far  as 
the  use  of  time  was  concerned  it  was  found  that  information  processing  was  speeded  up,  but 
subjects  waited  relatively  the  same  amount  of  time  before  they  started  to  intervene  with  the  system. 

Notes: 

This  article  details  a  study  looking  at  the  macro  effects  of  time  pressure  on  a  dynamic 
attendant/athlete  scenario.  Subjects  monitored  the  simulated  athlete’s  fitness  level  through  a  graph 
on  a  computer  screen  which  ranged  from  0  (poor)  to  1 00  (good).  When  the  athlete’s  performance 
started  to  drop,  they  could  request  different  types  of  additional  information  (e.g.  tired?)  and 
perform  specific  actions  (e.g.  rest)  to  tiy  to  improve  the  athlete’s  condition.  Three  different  time 
horizons  (i.e.  slopes)  for  the  changing  graph  were  used  (long,  moderate  and  short). 

Results  showed  that  subjects  requested  the  same  amount  and  type  of  information,  no  matter  the 
time  horizon.  Subjects  performed  fewer  actions  in  the  more  restricted  time  horizon  and  made  less 
errors.  This  occurred  as  subjects  tended  to  act  more  (i.e.  apply  treatment)  when  there  was  more 
time,  and  judge  more  (i.e.  ask  for  more  information)  when  there  was  less  time. 

This  result  is  based  on  the  fact  that  in  this  particular  scenario  that  a  wrong  action  was  five  times  the 
cost  of  information  request.  In  other  scenarios  where  actions  are  less  “expensive”  the  authors 
anticipate  that  the  subjects  will  maintain  their  action-oriented  strategy  even  under  high  time 
pressure.  Though,  realistically,  in  most  DDM  environments  action  errors  will  be  costly. 

This  study  provides  some  interesting  evidence  about  the  effects  of  time  constraints  on  decision 
making.  The  study  seemed  to  err  on  the  side  of  having  enough  time  rather  than  pushing  subjects 
into  real  time  constraints  thus  the  role  of  mental  underload  is  suggested  as  a  contributing  factor  to 
the  poor  low  time  pressure  results.  In  general,  the  article  does  not  talk  specifically  about  cues  or 
pattern  recognition,  but  their  use  can  be  inferred. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  CUES  and  PATTERN  RECOGNITION. 

To  be  used?  FOR  CONSIDERATION 
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Author(s):  Kirlik,  A.,  Walker,  N.,  Fisk,  A.  D„  &  Nagel,  K. _ 

Source:  Human  Factors,  38,  288-299 

Year:  1996 _ 

Abstract: 

Skilled  performers  in  complex  environments  rely  heavily  on  heuristic  strategies  to  cope  with  the 
time  pressure  and  complexity  of  dynamic  tasks.  We  suggest  that  the  use  of  task  simplification 
strategies  based  largely  on  perception  and  pattern  recognition  is  fundamental  to  the  novice-expert 
shift  in  dynamic  decision  making.  We  therefore  suggest  that  interface  training  interventions  should 
support  the  development  of  highly  effective  and  robust  heuristic  strategies,  rather  than  the 
development  of  more  abstract,  cognitively  intensive  strategies.  A  pair  of  empirical  studies  are 
presented  that  investigated  the  benefits  of  training  interventions  aimed  at  supporting  perceptual  and 
pattem-recognitional  activities  in  dynamic  environments.  Results  suggest  that  the  acquisition  of 
skilled  performance  in  dynamic  environments  can  be  accelerated  by  supporting  perceptual 
activities  in  the  service  of  dynamic  decision  making.  Implications  of  these  results  for  training, 
aiding,  and  interface  design  are  discussed. 

Notes: 

Kirlik  et  al.  investigate  particular  coping  strategies  in  dynamic  decision  making  tasks.  Using 
heuristics  may  be  a  common  coping  strategy.  The  use  of  these  heuristic  rely  largely  on  perceptual 
information  and  pattern  recognition,  and  less  so  on  abstract,  analytic  strategies. 

Serious  errors  have  been  demonstrated  when  people  rely  on  inappropriate  perceptual  information. 

The  authors  conduct  two  studies  to  look  at  the  use  of  perceptual  information  in  DDM.  The  first 
focuses  on  comparing  participant  performance  between  those  who  have  received  training  which 
focuses  on  their  abstract  rule  learning  and  visual  enhancements  as  techniques  to  support  DDM. 

Two  factors  were  examined:  rule  learning  vs.  no  rule  learning  and  visual  enhancement  display  vs. 
no  enhancement  display. 

Learning  was  conducted  by  having  participants  read,  memorize,  and  recall  all  rules.  Information  on 
the  digital  display  was  enhanced  by  highlighting  the  critical  information  on  the  screen.  The  context 
of  this  task  was  a  simulated  football  game  where  the  participants  had  to  identify  when  the  receiver 
came  open. 

Results  showed  that  the  two  groups  that  received  rule-based  training  were  equally  accurate; 
however,  the  group  that  received  rule-based  training  and  visual  enhancement  training  was  superior 
in  response  speed.  The  group  that  had  rule  training  and  no  visual  enhancement  had  the  slowest 
response  speed.  This  shows  the  importance  of  having  training  on  both  the  abstract  knowledge 
components  and  on  concrete  aspects  of  the  task.  The  time  difference  may  be  due  to  the  heavier 
reliance  on  abstract  reasoning  versus  perceptual  and  pattern-recognition  heuristics. 

The  second  study  investigated  the  use  of  the  visual  display  enhancement  in  support  of  DDM 
behaviour.  In  this  study  the  information  highlighted  pertained  to  the  critical  properties  and 
relationships  in  the  DDM  task.  The  context  of  this  task  was  a  war  where  participants  commanded  a 
set  of  UAVs  (Unmanned  Arial  Vehicles)  in  an  offensive  attack.  This  study  included  a  manipulation 
of  the  level  of  workload. 

Results  showed  that  two  display  groups  performed  equally  well  under  normal  conditions;  however, 
in  the  increased  workload  conditions  participants  who  had  the  perceptual  enhanced  display  had 
better  performance,  indicating  they  acquired  more  robust  skills  than  those  using  the  baseline 
display. 
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Conclusions  were  that  perceptual  augmentation  1)  facilitates  decision-making  skill  acquisition;  2) 
results  in  more  robust  performance  in  situations  with  higher  workloads;  and  3)  its  removal  creates 
no  degradation  of  skilled  performance. 

The  authors  investigate  decision  making  using  complex,  dynamic  environments,  though  in  a 
laboratory-based  setting  they  are  able  to  control  identifying  information  and  highlight  valid  cues 
which  is  not  practical  nor  feasible  in  real-life  DDM. 

Challenges  in  DDM  when  using  heuristics  enter  at  the  level  of  perception  and  discrimination. 
Identifying  critical  information  within  the  environment,  and  having  rule-based  training  can 
improve  DDM  performance,  decision  times,  and  robust  DDM  skills.  How  this  can  be  done,  is 
through  enhancement  of  valid  perceptual  information  that  has  direct  relationship  to  the  criterion. 
This  may  not  be  feasible  in  uncertain  DDM  tasks  where  there  is  no  overseer  who  knows  what 
information  is  relevant  to  the  decisions  that  need  to  be  made  (e.g.,  military  wargaming). 

This  article  shed  light  on  the  feasibility  of  developing  training  principles  to  “foster  efficient  access 
to,  and  use  of,  critical  perceptual  information  in  DDM  environment”. 

Training  in  DDM  then  may  focus  on  helping  participants  to  gain  skills  in  identifying  information, 
as  it  is  shown  in  this  study  after  taking  away  perceptual  augmentation  (highlighting  the 
information)  the  performance  is  not  diminished.  How  transferable  this  may  be  to  dissimilar 
situations  may  be  in  question. 

Using  rule-based  training,  basically  to  train  a  particular  type  of  heuristics  or  “short-cuts”,  in 
decision  making  were  shown  to  improve  DDM  performance. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  CUE  INTEGRATION 
To  be  used?  INCLUDE  IF  RELEVANT 


42.  Title:  Recognition-Primed  Decision  Strategies 
Author(s):  Klein,  G.  &  Crandall,  B. 

Source:  Report  prepared  for  the  United  States  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  ARI  Report  No.  96-36. 

Year:  1996 

Abstract: 

We  describe  activities  conducted  during  a  3-year  basic  research  contract  that  has  its  goal  extension 
and  examination  of  a  Recognition-Primed  Decision  (RPD)  model  of  decision  making.  The  RPD 
model  describes  a  decision  strategy  commonly  employed  by  proficient  personnel  called  upon  to 
make  decisions  in  operational  settings  by  high  risk,  time  constraints,  and  ambiguous  or  incomplete 
information.  Work  was  organized  around  three  areas  of  interest:  (1)  evaluation  of  the  relative 
strengths  and  weaknesses  of  the  RPD  strategy,  (2)  examination  of  the  nature  of  simulation 
assessment  in  C2  environments,  and  (3)  exploration  of  techniques  for  supporting  decision  making 
in  operational  environments  that  are  consistent  with  the  RPD  framework.  The  report  contains 
descriptions  of  nine  studies  (seven  empirical,  two  analytical)  conducted  to  examine  these  issues. 
One  important  outcome  of  the  research  has  been  to  advance  understanding  of  the  role  of  mental 
simulation  in  decision  making.  We  have  developed  a  detailed  mode  of  mental  simulation,  including 
an  account  of  how  mental  simulation  serves  as  a  source  of  power  for  a  variety  of  cognitive 
functions.  Work  performed  under  this  contract  has  helped  to  establish  naturalistic  decision  making 
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as  an  important  and  unique  perspective. 

Notes: 

Klein  is  a  critical  player  in  the  natural  decision  making  domain,  his  major  contribution  being  the 
Recognition  Primed  Decision  Making  (RPDM)  model.  Detailing  the  creation  process  of  the  RPDM 
model,  this  report  guides  the  reader  through  initial  and  subsequent  phases  of  RPDM  models.  This 
model  challenges  the  application  of  classic  decision  making  models  by  experts  under  complex, 
ambiguous  and  time  constrained  scenarios.  This  model  was  developed  after  observing  various 
expert  decision  makers  in  their  operational  environments  (e.g.  fire  fighters).  Many  of  these 
environments  require  dynamic  decision  making.  This  report  details  the  development  of  the  RPDM 
model  and  expands  it  from  its  initial  version.  Biases  due  to  heuristics  (of  which  experts  are  just  as 
susceptible  as  novices)  are  also  discussed  in  relation  to  the  RPDM  model. 

The  RPDM  model  suggests  that  experienced  decision  makers  are  able  “to  recognize  the  essential 
characteristics  of  a  situation”  (p.  2)  to  identify  goals  and  possible  action  paths.  Possible  actions  are 
then  evaluated  serially  until  a  satisfactory  -  but  not  necessarily  optimal  -  course  of  action  is 
determined.  Three  cases  of  RPDM  are  included,  which  vary  in  the  level  of  mental  evaluation  or 
simulation  that  the  course  of  action  is  subjected  to.  The  mental  simulation  process  is  discussed  in 
detail  followed  by  the  situation  assessment  process.  Cue  integration,  pattern  recognition  and 
heuristics  play  distinct  roles  in  these  processes. 

Limitations  of  the  RPD  model  are  included  in  the  report  (e.g.  mental  simulation  requires  a  great 
deal  of  attention  and  can  be  easily  interrupted).  Looking  at  the  errors  that  arise  in  using  this  model, 
the  authors  found  the  most  common  errors  were  a  lack  of  experience,  lack  of  critical  information  or 
faulty  mental  simulation.  No  errors  due  to  poor  estimation,  improper  weighting  of  criteria  or 
confirmation  bias  were  found;  though,  mental  simulation  is  believed  to  lead  to  overconfidence. 
Training  in  mental  simulation  is  suggested  as  a  feasible  way  to  improve  decision  making. 

Finally,  team  decision  making  processes  are  discussed,  and  the  Advance  Team  Decision  Making 
model  is  laid  out. 

Though  this  article  uses  the  terms  “cue  integration”  and  “pattern  recognition”  rarely,  there  are  clear 
links  to  these  concepts  using  more  general  terms  throughout  various  stages  of  the  RPDM  model. 
This  is  a  highly  relevant  report  to  the  goals  of  this  project. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  CUES,  PATTERN  RECOGNITION, 
HEURISTICS,  AND  BIASES. 

To  be  used?  INCLUDE  IF  RELEVANT 


43.  Title:  Fast,  frugal,  and  fit:  Simple  heuristics  for  paired  comparison 
Author(s):  Martignon,  L.  &  Hoffrage,  U. 

Source:  Theory  and  Decision,  52,  29-71. 

Year:  2002 
Abstract: 

This  article  provides  an  overview  of  recent  results  on  lexicographic,  linear,  and  Bayesian  models 
for  paired  comparison  from  a  cognitive  psychology  perspective.  Within  each  class,  the  authors 
distinguish  subclasses  according  to  the  computational  complexity  required  for  parameter  setting. 

They  identify  the  optimal  model  in  each  class,  where  optimality  is  defined  with  respect  to _ 
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performance  when  fitting  known  data.  Although  not  optimal  when  fitting  data,  simple  models  can 
be  astonishingly  accurate  when  generalizing  to  new  data.  A  simple  heuristic  belonging  to  the  class 
of  lexicographic  models  is  Take  The  Best  (G.  Gigerenzer  &  D.  G.  Goldstein,  1996).  It  is  more 
robust  than  other  lexicographic  strategies  which  use  complex  procedures  to  establish  a  cue 
hierarchy.  In  fact,  it  is  robust  due  to  its  simplicity,  not  despite  it.  Similarly,  Take  The  Best  looks  up 
only  a  fraction  of  the  information  that  linear  and  Bayesian  models  require;  yet  it  achieves 
performance  comparable  to  that  of  models  which  integrate  information.  Due  to  its  simplicity, 
frugality,  and  accuracy,  Take  The  Best  is  a  plausible  candidate  for  a  psychological  model  in  the 
tradition  of  bounded  rationality.  The  authors  review  empirical  evidence  showing  the  descriptive 
validity  of  fast  and  frugal  heuristics. 

Notes: 

The  authors  review  models  for  paired  comparison  (e.g.,  choosing  1  of  2  options)  from  several 
domains.  Computational  complexity  is  one  of  the  key  variables  examined;  the  authors  report  that  a 
simple  heuristic,  “take  the  best”  (also  talked  about  by  Bryant  and  probably  others)  is  extremely 
simple  yet  surprisingly  robust,  and  adequately  allows  the  application  of  knowledge  to  new  data. 

The  authors  cite  Gigerenzer,  and  the  idea  that  the  mind  has  a  toolbox  of  fast,  frugal,  and  accurate 
heuristics  which  can  be  used  in  different  situations.  The  authors  discuss  three  types  of  models; 
lexographic,  linear,  and  Baysian. 

The  lexicographic  approach  for  determining  which  of  a  pair  of  options  has  a  higher  criterion  value 
involves  examining  the  cues  in  a  predetermined  order.  The  “take  the  best”  involves  examining  only 
the  most  valid  cue;  other  options  are  to  order  cues  either  optimally  or  based  on  some  aspect  of 
conditional  validity.  Note  that  these  models  may  stop  examining  cues  once  a  particular  level  of 
information  has  been  obtained. 

Another  approach  is  a  linear  approach;  in  this  approach,  all  cues  are  examined.  What  sets  this 
approach  apart  is  its  emphasis  on  cue  weights,  and  there  are  different  strategies  for  determining 
these  weights. 

Baysian  networks  compare  the  cue  profiles  and  determine  which  option  is  more  likely  to  exceed 
the  criterion  (which  has  a  larger  criterion  value).  This  is  closest  to  pattern  matching. 

The  authors  cite  evidence  that  people  tend  to  make  inferences  consistent  with  lexographic  models 
(e.g.,  take  the  best)  when  time  is  short  or  it  is  otherwise  costly  to  examine  many  cues.  Take  the  best 
may  also  be  the  basis  of  different  heuristics  and  biases,  including  the  hindsight  bias  and 
overconfidence. 

Note  that  these  decisions  are  not  DDM,  but  considering  the  requirements  in  DDM  that  decision 
making  be  fast  and  accurate,  it  is  likely  that  strategies  such  as  take  the  best  would  be  useful  in 
DDM.  Understanding  that  these  strategies  are  useful,  and  how  they  can  fall  short,  would  be 
beneficial  in  designing  training  programs  for  DDM. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  CUE 
INTEGRATION 
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Source:  Journal  of  Experimental  Psychology:  Learning,  Memory,  and  Cognition  30,  923-935 
Year:  2004 _ 

Abstract: 

In  2  experiments,  the  authors  sought  to  distinguish  between  the  claim  that  recognition  of  an  object 
is  treated  simply  as  a  cue  among  others  for  the  purposes  of  decision  making  in  a  cue-learning  task 
from  the  claim  that  recognition  is  attributed  a  special  status  with  fundamental,  noncompensatory 
properties.  Results  of  both  experiments  supported  the  former  interpretation.  When  recognition  had 
a  high  predictive  validity,  it  was  relied  on  (solely)  by  the  majority  of  participants;  however,  when 
other  cues  in  the  environment  had  higher  validity,  recognition  was  ignored,  and  these  other  cues 
were  used.  The  results  provide  insight  into  when,  where,  and  why  recognition  is  used  in  decision 
making  and  also  question  the  elevated  status  assigned  to  recognition  in  some  frameworks 

Notes: 

The  authors  examine  the  influence  of  recognition  -based  information  in  a  simple  decision-making 
task.  The  focus  of  Experiment  1  was  to  investigate  if  recognition  information  (in  this  study  a 
company-name  cue)  was  used  similarly  to  other  cue  information.  Experiment  2  tested  the  notion 
(based  on  a  statement  by  Goldstein  and  Gigerenzer,  2002)  that  recognition  information,  when 
presented,  was  used  as  a  sole  information  source  and  people  do  not  seek  further  information. 

Experiment  1 : 

Used  an  investment  scenario  to  investigate  how  recognition  information  was  used  in  a  decision¬ 
making  task.  Participants  were  presented  with  two  companies  in  which  they  could  invest.  Some 
participants  were  able  to  see  company  names  as  information  and  buy  advice  from  advisors.  Three 
conditions  were  used.  In  all  conditions  participants  received  advice  from  3  advisors  on  whether  to 
invest  in  a  particular  company  or  not.  The  two  conditions  were:  Recognition  High  (RH)  where  the 
company  name  had  the  highest  validity  (degree  to  which  the  cue  predicts  the  outcome)  and  highest 
discrimination  rate  (proportion  of  occasions  on  which  a  cue  has  a  different  values  for  each 
alternative)  making  it  the  best  source  of  information;  Recognition-Low  (RL)  condition  where  the 
company  name  cue  was  the  lease  valid  and  had  low  discrimination;  no  recognition  condition  where 
the  company  name  cue  was  replaced  by  advice  from  a  advisor. 

Equations  were  presented  that  could  be  used  to  determine  which  strategies  were  used  based  on  the 
payoff  (results)  achieved. 

The  usefulness  of  each  cue  was  rated  and  it  was  seen  that  within  the  RH  condition  the  company 
name  was  recognized  as  being  more  useful  than  any  of  the  advice.  There  were  large  individual 
differences  in  the  amount  of  extra  advice  purchased. 

Results  support  the  idea  that  recognition  cues  are  treated  in  a  similar  way  as  other  cues.  The 
validity  of  the  cues  was  the  primary  factor  in  determining  if  it  would  be  used  as  the  main 
information  in  the  decision-making  process. 

Experiment  2: 

The  RL  condition  was  used  in  Experiment  2  in  order  to  answer  the  question:  will  participants  learn 
to  use  advice  cues,  or  does  the  use  of  recognition  information  have  a  :special  status”  that  will 
override  the  temptation  to  search  for  and  use  additional  information? 

Results  indicate  that  the  majority  of  participants  still  did  purchase  and  followed  additional 
information  even  when  recognition  information  could  be  relied  on  solely.  Participants  may  have 
regarded  the  recognition  information  as  somewhat  special  as  they  tended  to  overestimate  the _ 
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usefulness  of  recognizing  the  company  name  for  making  accurate  decisions. 

The  overall  discussion  reiterates  the  findings  that  recognition-information  is  treated  similarly  to 
other  cue  information,  and  that  it  may  not  be  treated  as  uniquely  as  suggested,  though  participants 
often  overestimated  its  usefulness.  The  authors  discuss  the  findings  and  claims  of  the 
noncompensatory  use  of  recognition  information  (simple  recognition  overrides  alternatives  to 
search  for  more  information  about  there  recognized  object)  as  proposed  by  Goldstein  & 

Gigerenzer,  2002.  An  idea  is  discussed  in  which  one  process  gives  rise  to  an  automatic  generation 
of  familiarity  when  an  object  is  recognized,  and  another  process  assesses  the  basis  for  the  feeling  of 
familiarity  (Kahneman  &  Frederick,  2002). 

Not  dynamic  decision  making,  only  simple  decision-making  tasks  discussed  and  used  in  the 
experimentation. 

Within  an  investment  type  scenario,  recognition  cues  may  not  be  used  in  the  same  way  as  within 
other  decision-making  scenarios  because  it  may  be  considered  a  domain  in  which  multiple  sources 
of  information  should  be  consulted  prior  to  making  a  decision.  This  may  be  reflected  in  the  large 
individual  differences  in  the  amount  of  extra  advice  purchased 

Recognition  here  is  discussed  as  recognizing  objects,  not  patterns  or  multiple  pieces  of  information 
as  we  have  defined  it. 

The  discussion  about  different  theories  of  recognition  may  transfer  to  our  definition  of  pattern 
recognition  and  could  be  useful  to  the  current  project,  though  keeping  in  mind  that  all  approaches 
were  discussed  using  simple  decision  tasks.  Results  would  not  be  expected  to  transfer  well  to 
PPM. _ 

CAN  BE  USEP  TO  SUPPORT  INVESTIGATING:  PATTERN  RECOGNITION,  CUE 
INTEGRATION  (some  heuristics) 

To  be  used?  OO  NOT  USE. 


45.  Title:  On  the  psychology  of  the  recognition  heuristic:  Retrieval  primacy  as  a  key  determinant  of 
its  use 

Author(s):  Pachur,  T.  &  Hertwig,  R. 

Source:  Journal  of  Experimental  Psychology:  Learning,  Memory,  and  Cognition,  32, 983-1002. 
Year:  2006 _ 

Abstract: 

The  recognition  heuristic  is  a  prime  example  of  a  boundedly  rational  mind  tool  that  rests  on  an 
evolved  capacity,  recognition,  and  exploits  environmental  structures.  When  originally  proposed,  it 
was  conjectured  that  no  other  probabilistic  cue  reverses  the  recognition-based  inference  (P.  G. 
Goldstein  &  G.  Gigerenzer,  2002).  More  recent  studies  challenged  this  view  and  gave  rise  to  the 
argument  that  recognition  enters  inferences  just  like  any  other  probabilistic  cue.  By  linking 
research  on  the  heuristic  with  research  on  recognition  memory,  the  authors  argue  that  the  retrieval 
of  recognition  information  is  not  tantamount  to  the  retrieval  of  other  probabilistic  cues. 

Specifically,  the  retrieval  of  subjective  recognition  precedes  that  of  an  objective  probabilistic  cue 
and  occurs  at  little  to  no  cognitive  cost.  This  retrieval  primacy  gives  rise  to  2  predictions,  both  of 
which  have  been  empirically  supported:  Inferences  in  line  with  the  recognition  heuristic  (a)  are 
made  faster  than  inferences  inconsistent  with  it  and  (b)  are  more  prevalent  under  time  pressure. 
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Suspension  of  the  heuristic,  in  contrast,  requires  additional  time,  and  direct  knowledge  of  the 
criterion  variable,  if  available,  can  trigger  such  suspension. 

Notes: 

The  recognition  heuristic  refers  to  the  strategy  of  assuming  that  if  one  of  two  objects  is  recognized 
and  the  other  is  not,  then  the  recognized  object  has  the  higher  value  with  respect  to  the  criterion 
(e.g.,  is  more  frequent,  more  famous,  etc.). 

The  authors  attempt  to  address  the  question  of  why  humans  have  so  many  resources  allocated  to 
lower-level  tasks  such  as  perception  and  so  many  more  limitations  on  higher-level  cognitive 
functions.  The  authors  argue  that  complex  and  effective  cognition  can  arise  from  co-opting  more 
simple  functions,  and  that  appropriate  behaviour  is  the  result  of  selecting  the  right  tools  for  the 
right  situation,  rather  than  having  a  single,  large-capacity  cognitive  processor. 

The  authors  review  the  debate  about  whether  the  recognition  heuristic  is  compensatory  or 
noncompensatory  -  that  is,  whether  recognition  information  is  integrated  with  other  information 
when  making  a  judgment  (compensatory),  or  whether  it  overrides  other  information  and  drives  the 
judgment  (noncompensatory).  The  authors  review  and  critique  arguments  for  each  view,  and  argue 
that  recognition  information  is  special. 

The  authors  review  previous  research  and  provide  evidence  that  there  are  situations  in  which  the 
recognition  heuristic  is  not  used:  when  recognition  knowledge  did  not  evolve  naturally  (e.g.,  it  was 
derived  through  an  experiment)  or  when  recognition  is  traced  to  a  source  that  is  not  relevant  for  the 
criterion  variable,  and  recognition  is  used  as  a  heuristic  only  when  a  direct  solution  fails.  This  is  an 
important  indication  that  people  are  not  insensitive  to  conditions  in  which  the  heuristic  may  not  be 
appropriate,  and  do  not  use  it  if  they  have  other  options  which  would  be  more  accurate. 

The  authors  clarify  the  difference  between  the  recognition  and  availability  heuristics:  the 
availability  heuristic  involves  retrieving  instances  of  the  target  event,  whereas  the  recognition 
heuristic  involves  merely  recognizing  the  name  of  the  event  category. 

This  appears  to  be  an  article  with  well-performed  experiments,  although  some  of  their  data 
representation  seems  peculiar  (e.g.,  they  divided  RTs  into  percentiles  and  compared  differences 
within  those  percentiles,  which  I  haven’t  seen  done  before  and  is  not  justified  within  the  method). 

This  article  is  very  useful  in  exploring  how  heuristics  may  be  used,  how  they  might  rest  upon  more 
“basic”  perceptual  or  cognitive  processes,  and  how  they  may  be  “turned  off”  when  they  are  not 
appropriate.  Note  is  also  made  of  the  strong  individual  differences  in  heuristic  use.  This  article 
provides  a  useful  example  for  exploring  heuristic  use,  and  could  be  broadened  to  the  DDM  context 
(e.g.,  they  examine  the  use  of  the  heuristic  with  increased  time  pressure). 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  (some  CUE 
INTEGRATION) _ 
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Abstract: 

n/a 

Notes: 

Behavioural  Decision  Research  is  an  interdisciplinary  approach  that  explores  cognitive 
mechanisms  involved  in  decision  making  in  economics  and  other  complex  fields.  One  axiom  of  it 
is  that  preferences  and  beliefs  are  often  constructed,  and  are  dependent  not  just  on  the  task,  but  also 
the  decision  maker  and  the  context.  These  factors  lead  to  a  great  deal  of  variance  in  responses, 
going  against  common  sense  which  would  suggest  that  good  decisions  are  consistent.  When 
decision  makers  face  problems  where  one  choice  is  transparently  dominant,  consistent  responses 
are  found;  however,  dominance  can  be  easily  masked.  Framing  effects,  information  presentation, 
time  pressure,  scenario  complexity,  number  of  alternatives,  incentives,  uncertainty  or  ambiguity, 
experience  and  affect  decision  outcomes.  These  and  other  factors  are  discussed. 

Additional  cognitive  challenges  discussed  include:  evaluating  the  utilities  of  future  consequences, 
the  neglect  of  potentially  relevant  problem  information,  ignoring  values  when  they  are  difficult  to 
estimate,  “general  tendency  to  accept  problems  as  given  and  the  segregation  of  the  decision 
problem  at  hand  from  its  broader  context”  (p  1 16). 

Attributes  and  their  cognitive  structure  are  discussed,  providing  potential  insights  into  cue 
integration.  For  example,  “people  actively  restructure  decision  problems  until  one  alternative  is 
seen  to  be  dominant...  The  restructuring  may  involve  such  operations  as  collapsing  two  or  more 
attributes  into  a  more  comprehensive  one,  emphasizing  an  attribute  or  adding  new  attributes  to  the 
problem  representation  that  will  bolster  one  alternative”  (p  101).  Congruence  between  the  format 
of  the  information  and  judgement  required  is  also  suggested  to  affect  cue  integration.  Though  how 
this  congruence  is  determined  was  not  clearly  described. 

Briefly,  a  continuum  of  decision  making  from  intuitive  to  rational  is  detailed.  Interestingly,  the 
authors  suggest  that  errors  will  be  fewer  but  larger  after  rational  analysis  compared  to  intuitive 
judgments. 

Some  of  the  studies  discussed  do  not  relate  directly  to  DDM  scenarios,  but  in  general  the  article  is 
looking  at  decision  making  under  complex  and  uncertain  circumstances. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  CUES,  HEURISTICS,  and  BIASES. _ 

To  be  used?  INCLUDE  IF  RELEVANT 


47.  Title:  A  model  of  human  decision  making  in  complex  systems  and  its  use  for  design  of  system 
control  strategies 

Author(s):  Rasmussen,  J.  &  Lind,  M. 

Source:  Paper  presented  at  the  American  Control  Conference,  June  14-16,  Arlington,  Virginia, 
U.S.A. _ 

Year:  1982 

Abstract: 

The  paper  describes  a  model  of  operators'  decision  making  in  complex  system  control,  based  on 
studies  of  event  reports  and  performance  in  control  rooms.  This  study  shows  how  operators  base 
their  decisions  on  knowledge  of  system  properties  at  different  levels  of  abstraction  depending  on 
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their  perception  of  the  system's  immediate  control  requirements.  These  levels  correspond  to  the 
abstraction  hierarchy  including  system  purpose,  functions,  and  physical  details,  which  is  generally 
used  to  describe  a  formal  design  process.  In  emergency  situations  the  task  of  the  operator  is  to 
design  a  suitable  control  strategy  for  systems  recovery,  and  the  control  systems  designer  should 
provide  a  man-machine  interface,  supporting  the  operator  in  identification  of  his  task  and  in 
communication  with  the  system  at  the  level  of  abstraction  corresponding  to  the  immediate  control 
requirement.  A  formalized  representation  of  system  properties  in  a  multilevel  flow  model  is 
described  to  provide  a  basis  for  an  integrated  control  system  design. 

Notes: 

It  is  pointed  out  that  there  are  many  subjective  aspects  to  a  decision  (e.g.,  information  known, 
preferences,  and  circumstances)  within  an  unfamiliar  situation.  It  may  not  be  realistic  to  model  the 
detail  flow  of  information  processes  within  this  type  of  decision. 

Rasmussen  &  Lind  identify  the  states  of  knowledge  within  a  generic  decision.  Decision  Ladder 
structures  are  formulated.  Shortcuts  can  be  used  to  skip  steps  within  the  DL  to  make  quicker 
decisions.  Skill-based,  rule-based,  and  knowledge-based  human  performance  is  examined  and  how 
they  demonstrate  different  levels  of  automated  or  analysis-based  action. 

The  authors  propose  that  system  control  requires  systematic  state  identification  (identification  of 
any  changes  or  deviations  from  a  specified  norm)  by  means  of  “logical  inferences  based  on 
measurable  variables.”  [comment:  not  possible  in  military  DDM\ 

Decision  tasks  to  operators  and  computers  are  considered.  Overall  control  structure  and  task 
allocations  are  developed  in  terms  of  an  abstract-flow  typology. 

The  decision  ladder  structure  identifies  knowledge  states  and  stages  within  a  decision.  Relating 
these  to  cognitive  tasks  within  each  state/stage  could  be  a  method  to  identify  types  of  challenges. 
Dynamic  decisions  would  need  to  be  the  focus-  this  may  require  some  additional  interpretation  and 
inferences. 

The  “leaps”  and  “shunts”  that  act  as  bypasses  from  one  rung  on  the  ladder  to  another  use  heuristics 
to  skip  stages  in  the  decision  making  process. 

The  authors  identify  the  difficulty  in  identifying  information  flow  within  new,  unfamiliar  situations 
in  a  decision  making  task. 

Control  of  a  situation  is  derived  from  the  relationships  between  the  actual  state,  desired  state  (or 
changes  of  states)  and  required  actions.  In  addition  if  we  include,  external  factors  that  influence 
changes  within  a  state,  or  changes  to  information  this  can  be  representative  of  DDM. 

The  background  of  the  decision  maker  may  influence  the  conceptual  framework  of  the  decisions. 

The  authors  further  propose  that  experts  are  able  to  identify  required  information  in  analyzing  the 
situation,  and  ignore  irrelevant  information,  or  determine  if  information  is  relevant  after  it  is 
examined,  and  switch  to  an  effective  strategy  of  information  searching  when  required.  To  do  this 
one  must  have  a  high  capacity  for  remembering  required  information.  This  could  be  a  challenge  - 
particularly  under  high  stress. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  general  DM  structure,  HEURISTICS 
To  be  used?  INCLUDE  IF  RELEVANT 
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Author(s):  Rodgers,  R.  S. 

Source:  Report  prepared  for  the  Air  Force  Research  Laboratory,  Report  No.  AFB  OH  45433-7604. 

Year:  2006 

Abstract: 

Intelligence  analysts  and  mental  health  clinicians  have  some  aspects  of  their  respective  crafts  in 
common.  In  many  cases  both  have  to  make  predictions  about  fliture  behavior.  Findings  from  the 
clinical  literature  were  used  to  make  the  point  that  humans,  in  general,  are  not  particularly  skilled  at 
combining  various  pieces  of  information  in  order  to  make  predictions,  and  by  extension, 
intelligence  analysts  suffer  the  same  fate.  Understanding  the  problems  involved  in  information 
processing  can  help  us  develop  methods  and  tools  to  assist  in  mitigating  three  broad  cognitive 
errors:  (a)  the  tendency  to  see  patterns  where  none  exist,  (b)  the  tendency  to  seek  confirmatory 
evidence,  and  (c)  the  use  of  preconceived  biases. 

Notes: 

Focusing  on  similarities  between  the  tasks  of  clinicians  and  intelligence  analysts,  the  author  looks 
at  tasks  with  poor  performance  and  potential  underlying  causes.  For  example,  neither  group 
appears  to  perform  prediction  tasks  effectively,  they  also  have  limited  perception  abilities,  seek 
confirmatory  evidence  and  show  preconceived  biases.  It  is  noted  that  although  decision  maker  may 
strive  for  accurate  results,  100%  correctness  is  not  a  realistic  goal  in  either  workspace.  The  body  of 
the  report  looks  at  classic  heuristics  (availability,  overconfidence,  representativeness,  anchoring 
and  adjustment,  and  hindsight)  and  details  scenarios  where  these  biases  can  manifest  themselves. 
Finally,  decision  making  recommendations  are  included. 

This  article  provides  a  succinct  review  of  various  classical  heuristics  and  how  they  impact  the  work 
of  both  clinicians  and  intelligence  analysts.  The  extent  to  which  either  of  these  environments 
involve  dynamic  decision  making  is  subjective.  This  article  does  not  offer  much  in  terms  of 
insights  into  biases  or  DDM. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  HEURISTICS  AND  BIASES. _ 

To  be  used?  FOR  CONSIDERATION 


49.  Title:  Bias  in  planning  and  explanation-based  learning 
Author(s):  Rosenbloom,  P.  S.,  Lee,  S.,  &  Unruh,  A. 

Source:  Report  prepared  for  the  Defense  Advanced  Research  Projects  Agency,  Arlington,  VA, 
ARPA  Report  No.  AD-A269  608. _ 

Year:  1993 

Abstract: 

Biases  enable  systems  to  make  decisions  in  realms  where  all  legitimate  sources  of  knowledge  have 
been  exhausted.  This  article  investigates  the  application  of  biases  to  the  problem  of  planning,  and 
how  this  can  indirectly  induce  effective  biases  in  a  learning  process  that  is  based  on  planner's 
experiences.  Experimental  results  from  six  biased  planners,  plus  several  more  complex 
multimethod  planners,  indicate  complex  tradeoffs  among  planner  completeness,  planning 
efficiency,  and  length.  Learning  also  varies  in  complex  ways  among  these  planners,  with  one 
notable  result  being  the  ease  with  which  some  planners  learn  rules  that  can  generalize  from  one 
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object  to  many;  a  phenomenon  known  in  machine  learning  as  generalization  to  N. 

Notes: 

The  authors  are  interested  in  how  bias  is  used  to  make  planning  decisions  when  there  is  inadequate 
information,  and  how  these  biases  affect  learning.  According  to  the  authors,  bias  determines  the 
answer,  but  search  control  is  what  determines  the  efficiency  with  which  the  answer  is  found. 

Biases  are  beneficial  in  several  ways,  including: 

•  Biases  can  help  organize  and  understand  concepts 

•  Can  reduce  computational  effort  (reduce  number  of  COAs  to  be  examined) 

•  Can  create  effective  bias  on  learning 

Absolute  biases  are  biases  which  reduce  the  problem  space;  these  biases  must  be  used  properly  or 
they  will  result  in  faulty  answers.  Relative  biases  simply  create  preferences  in  solutions,  they  don’t 
eliminate  solutions  (part  of  the  problem  space). 

Experiments  were  performed  using  a  software  program  (SOAR)  to  investigate  these  benefits. 

It  is  unknown  whether  people  actually  use  these  biases,  as  no  human  data  was  provided,  and  there 
is  not  enough  discussion  of  how  people  would  or  could  actually  use  these  biases.  However,  the 
discussion  of  how  biases  can  be  useful  in  creating  plans  and  learning  the  planning  process  could 
add  to  this  project. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  BIASES 
To  be  used?  FOR  CONSIDERATION 


50.  Title:  Inferring  rule-based  strategies  in  dynamic  judgment  tasks:  Towards  a  noncompensatoiy 
formulation  of  the  lens  model 

Author(s):  Rothrock,  L.  &  Kirlik,  A. 

Source:  Report  prepared  for  the  Naval  Training  Systems  Center,  Orlando,  FL.  Report  No.  AHFD- 
03-5/NTSC-03-I. _ 

Year:  2003 
Abstract: 

Performers  in  time-stressed,  information-rich  tasks  develop  rule-based,  simplification  strategies  to 
cope  with  the  severe  cognitive  demands  imposed  by  judgment  and  decision  making.  Linear 
regression  modeling,  proven  useful  for  describing  judgment  in  a  wide  range  of  static  tasks,  may 
provide  misleading  accounts  of  these  heuristics.  That  approach  assumes  cue-weighting  and  cue- 
integration  are  well  described  by  compensatory  strategies.  In  contrast,  evidence  suggests  that 
heuristic  strategies  in  dynamic  tasks  may  instead  reflect  rule-based,  noncompensatory  cue  usage. 
We  therefore  present  a  technique,  called  Genetics-Based  Policy  Capturing  (GBPC),  for  inferring 
noncompensatory,  rule-based  heuristics  from  judgment  data,  as  an  alternative  to  regression.  In 
GBPC,  rule-base  representation  and  search  uses  a  genetic  algorithm,  and  fitting  the  model  to  data 
uses  multi-objective  optimization  to  maximize  fit  on  three  dimensions:  a)  completeness  (all  human 
judgments  are  represented);  b)  specificity  (maximal  concreteness);  and  c)  parsimony  (no 
unnecessary  rules  are  used).  GBPC  is  illustrated  using  data  from  the  highest  and  lowest  scoring 
participants  in  a  simulated  dynamic,  combat  information  center  (CIC)  task.  GBPC  inferred  rule- 


Page  124 


Cue  integration  in  Dynamic  Decision  making 


Human systems 


HUMAHSYSTEMS 

ittfp*nu4 


bases  for  these  two  performers  that  shed  light  on  both  skill  and  error.  We  compare  the  GBPC 
results  with  regression-based  Lens  Modeling  of  the  same  data  set,  and  discuss  how  the  GBPC 
results  allowed  us  to  interpret  the  high  scoring  performer's  highly  significant  use  of  unmodeled 
knowledge  (C=l)  revealed  by  Lens  Model  analysis.  The  GBPC  findings  also  allow  us  to  now 
interpret  a  similarly  high  use  of  unmodeled  knowledge  (C=l)  in  a  previously  published  Lens 
Model  analysis  of  a  different  data  set  collected  in  the  same  experimental  task. 

Notes: 

The  authors  attempted  to  provide  feedback  to  participants  by  gathering  behavioural  data  and 
inferring  what  heuristics  people  were  using  to  make  decisions.  Thus,  feedback  can  be  tailored  to 
the  individual.  The  authors  argue  that  experienced  decision  makers  deal  with  time  pressure, 
uncertainty,  and  large  amounts  of  information  by  developing  heuristics  which  they  use  to 
oversimplify  the  information.  Like  most  heuristics,  these  will  sometimes  be  appropriate  and 
sometimes  lead  to  bias. 

The  lens  model  is  a  way  of  determining  how  someone  is  making  decisions  (i.e.,  what  criteria  they 
are  using  and  how  the  criteria  are  weighted);  however,  applying  this  model  requires  that  the 
judgment  environment  is  adequately  modeled  (appropriate  judgment  criteria  can  either  be 
objectively  measured  or  estimated  by  SMEs). 

The  authors  discuss  compensatory  and  noncompensatory  strategies.  Compensatory  strategies  are 
strategies  which  weigh  information  and  opposite  values  can  cancel  each  other  out.  As  well,  the 
presence  of  a  cue  with  high  value  (weight)  can  compensate  for  an  absence  of  cues  with  low  weight 
(e.g.,  evidence  for  the  decision  can  still  be  considered  strong).  Noncompensatory  strategies  are 
strategies  where  this  does  not  happen  (the  “trading  off’  property  doesn’t  apply;  e.g.,  every  cue 
must  have  a  high  value  for  evidence  for  the  decision  to  be  considered  strong).  The  authors  note  that 
many  heuristics  are  noncompensatory  strategies.  The  authors  note  that  noncompensatory  strategies 
often  yield  very  good  results  compared  to  more  cognitively  demanding  compensatory  strategies. 
The  authors  discuss  a  strategy  for  inferring  the  noncompensatory  heuristics  which  are  actually 
used. 

Note  that  the  authors  attempt  to  provide  empirical  evidence  for  their  theory;  however,  the 
participants  used  were  nonexperts.  As  the  theory  is  supposed  to  apply  to  experts  who  develop  their 
own  heuristics,  this  is  probably  not  appropriate  (although  it  says  they  were  trained  on  1 5  30-minute 
scenarios,  it  is  unclear  whether  they  reached  expert  level). 

One  proposed  heuristic  is  to  actively  search  for  diagnostic  information.  This  implies  the  ability  to 
learn  what  information  is  diagnostic,  and  that  it  is  available. 

This  is  an  article  that  is  somewhat  relevant  to  DDM;  based  on  the  amount  of  information  available, 
the  constantly  changing  nature  of  that  information,  and  the  time  pressure  that  is  often  on  people  to 
act  in  the  DDM  environment,  it  is  reasonable  to  suggest  that  heuristics  are  used  to  manage 
performance  in  DDM.  To  use  this  method  to  understand  behaviour  in  the  DDM,  performance 
would  have  to  be  monitored  in  realistic  situations,  but  the  “ground  truth”  would  have  to  be  known 
in  order  to  interpret  the  results.  As  well,  it  is  uncertain  the  benefit  that  can  be  gained  from 
providing  people  feedback  about  their  own  heuristics,  as  this  was  not  examined  in  this  paper. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  AND  CUE 
INTEGRATION 

To  be  used?  FOR  CONSIDERATION 
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51 .  Title:  Using  prospect  theory  to  investigate  decision-making  bias  within  an  information  security 
context. 

Author(s):  Schroeder,  N.  J. 

Source:  Master’s  thesis  submitted  to  the  Air  Force  Institute  of  Technology,  Wright-Patterson  Air 
Force  Base,  Ohio. 

Year:  2005 _ 

Abstract: 

Information  security  is  an  issue  that  has  increased  in  importance  over  the  past  decade.  In  this  time 
both  practitioner  and  academic  circles  have  researched  and  developed  practices  and  process  to 
more  effectively  handle  information  security.  Even  with  growth  in  these  areas  there  has  been  little 
research  conducted  into  how  decision  makers  actually  behave.  This  is  problematic  because  decision 
makers  in  the  Department  of  Defense  have  been  observed  exhibiting  risk  seeking  behavior  when 
making  information  security  decisions  that  seemingly  violate  accepted  norms.  There  are  presently 
no  models  in  the  literature  that  provide  sufficient  insight  into  this  phenomenon.  This  study  used 
Prospect  Theory  as  a  framework  to  develop  a  survey  in  an  effort  to  obtain  insight  into  how  decision 
makers  actually  behave  while  making  information  security  decisions.  The  survey  was  distributed  to 
Majors  in  the  Air  Force  who  represented  likely  future  information  security  decision  makers.  The 
results  of  the  study  were  mixed,  showing  that  prospect  theory  had  only  limited  explanatory  power 
in  this  context.  The  most  significant  finding  showed  that  negatively  connotated  decision  frames 
result  in  significantly  more  risk  seeking  behavior.  These  results  provide  insight  into  decision  maker 
behavior  and  highlight  the  fact  that  there  are  biases  in  information  security  decision  making. 

Notes: 

This  thesis  concerns  itself  with  the  security  of  information  and  how  decision  makers  are  handling 
it.  Prospect  Theory  developed  by  Kahneman  and  Tversky  is  used  as  a  framework  to  obtain  insight 
into  this  behaviour  using  a  survey  based  data  collection.  Note  that  a  limitation  to  using  survey- 
based  data  gathers  insight  only  to  the  intentional  behaviour  rather  than  actual  behaviour. 

The  first  step  in  the  research  agenda  is  to  better  understand  the  gap  between  CA’s  (Certification 
Authority)  and  DAA’s  (Designated  Approving  Authority)  in  the  DITSCAP  (Department  of 
Defence  Information  Technology  Security  Certification  and  Accreditation  Process).  Following  this 
the  research  will  investigate  if  there  are  biases  in  DM  that  influence  a  decision  maker  in  such  an 
information-security  context. 

A  literature  review  was  conducted  to  gather  background  on  information  security  processes  in 
businesses.  The  review  showed  that  research  has  lead  to  the  development  of  a  prescriptive 
approach  for  managing  information  security  and  provided  evidence  that  demonstrates  how  to  make 
decisions  in  this  area.  Research  efforts  have  also  explored  end-user  issues  (e.g.,  implementation  of 
technology  and  security  behaviour). 

Previous  research  on  decision  making  is  also  explored  in  attempts  to  identify  a  method,  or 
combination  of  methods  that  may  best  suit  the  context  of  information  security.  Research  looking 
into  decision  making  under  risk  also  within  this  scope  (usually  pertaining  to  a  business  context)  is 
examined.  It  is  concluded  that  there  is  minimal  prior  research  into  managerial  decision  making  in 
the  information-security  context. 

Based  on  the  literature.  Prospect  theoiy  was  chosen  as  a  model  that  meets  the  criteria  for  DM  under 
risk  and  DM  made  independent  of  dispositional  and  organizational  factors. 
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The  survey  study  conducted  assessed  1 1  hypotheses  that  investigated  the  use  of  Prospect  Theory  to 
model  decision  making  behaviour  in  information  security.  The  results  were  mixed.  Some 
hypothesis  were  supported,  others  were  not. 

A  summary  of  the  results  shows  that  negatively-connotated  decision  frames  result  in  significantly 
more  risk-seeking  behaviour. 

*H1 :  supported:  DM  are  significantly  risk  averse  in  general  information-security  gain  domains. 

H2:  not  supported:  conclusions  are  that  DM  are  indifferent  in  risk  behaviour  in  general 
information-security  loss  domains. 

H3:  not  supported:  conclusions  are  that  risk  seeking  behaviour  is  similar  between  decision  makers 
in  operationally-  and  security-framed  loss  domains. 

*H4:  supported:  decision  makers  are  more  risk  seeking  in  negatively-framed  information-security 
outcomes  than  in  the  similar  positively-framed  outcomes. 

H5a:  not  supported:  conclusions  are  that  decision  makers  did  not  behave  in  a  more  risk-averse  way 
when  there  was  a  shift  in  the  reference  point  into  a  loss  domain. 

H5b:  not  supported:  conclusions  are  similar  to  above,  decision  makers  did  not  behave  more  averse 
to  risk  for  significant  gains  when  there  was  a  shift  in  the  reference  point  into  a  loss  domain. 

*H6:  supported:  decision  makers  tend  to  give  a  greater  decision  weight  to  operations  outcome 
when  the  situation  presents  both  security  and  operations. 

*H7a:  supported:  Compared  with  operational  decision  makers,  non-operational  decision  makers  are 
significantly  more  averse  to  risk  in  information  security  gain  domains. 

H7b:  not  supported:  conclusions  explore  an  issue  similar  to  the  above,  though  in  a  loss  domain; 
Compared  with  operational  decision  makers,  non-operational  decision  makers  are  significantly 
more  averse  to  risk  in  information  security  loss  domains. 

H7c:  not  supported:  the  hypothesis  explored  investigates  if  in  an  information  security  context 
operational  decision  makers  prefer  operational  outcomes  to  security  outcomes  compared  with  non- 
operational  decision  makers. 

*H7d:  supported:  When  the  reference  point  is  shifted  negatively,  operational  decision  makers  are 
significantly  more  risk  avers  in  information  security  related  gain  domains  compared  with  non- 
operational  decision  makers. 

H7e:  not  supported:  Similar  to  H7d  within  a  loss  domain;  When  the  reference  point  is  shifted 
negatively,  operational  decision  makers  are  significantly  more  risk  avers  in  information  security 
related  loss  domains  compared  with  non-operational  decision  makers. 

(all  *  hypothesis  were  supported). 

Conclusions  from  these  results  imply  that  external  factors  (those  outside  the  control  of  the  decision 
maker)  can  influence  a  bias  in  decision  makers’  risk  behaviour  (based  on  H4).  It  is  also  suggested 
that,  in  general  (based  on  H6),  decision  makers  tend  to  lace  a  greater  weight  on  particular  pieces  of 
the  situation  (or  pieces  of  information,  e.g.,  operational  versus  security).  The  authors  showed 
evidence  that  Prospect  Theory  provided  insight  that  can  be  used  as  a  starting  point  for  future 
research,  though  does  not  provide  a  perfect  model  for  describing  decision  making  under  risk  in 
information  security  contexts. 

Limitations  to  this  study  include: 

Minimal  time  allotted  during  class  time  for  participants  to  complete. 
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The  study  measures  intentions  rather  than  actual  behaviour. 

Scenarios  used  were  at  the  researcher’s  discretion  and  wording  may  not  adequately 
represent  the  domain.  This  may  explain  some  reasons  why  some  Ho  were  not  confirmed. 

Questions  were  general,  and  more  specific  situations  may  provide  better  insight. 

Situations  explored  in  this  review  and  study  were  not  dynamic. 

The  author  focuses  on  the  description  of  background  information  and  of  Prospect  Theory  and  how 
it  can  be  implemented  to  DM  in  an  information  security  context. 

Literature  on  DM  is  reviewed  and  the  findings  are  shared. 

The  literature  review  presented  a  few  helpful  insights  into  challenges  in  normative  reasoning 
examples: 

Older  people  take  more  risk  to  avoid  loss  and  less  risk  for  gains. 

Bias:  A  “fads  and  fashion”  effect-  the  more  excitement  around  a  particular  industry,  the 
more  likely  managers  were  to  underestimate  risk. 

Expectations  interfere  with  risk  in  DM.  When  people  expect  their  choices  to  be  reviewed 
by  others  decision  makers  become  more  unwilling  to  take  risks. 

People  in  a  higher  position  of  authority  take  more  risk.  Societal  pressure  to  “make  things 
happen”  and  take  good  risks  to  do  so. 

People  are  often  unaware  of  the  influence  of  priming  to  risk  related  issues  affect  their  level 
of  risk  seeking  behaviour. 

Overall  conclusions  show  that  people  can  be  biased  to  external  factors  that  are  out  of  their  control, 
and  that  DMs  may  give  preference,  or  over  weighting,  to  some  information  rather  than  others 
influencing  their  final  decision. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  BIASES 
To  be  used?  FOR  CONSIDERATION 


52.  Title:  A  framework  for  military  decision  making  under  risks 
Author(s):  Schultz,  J.  V. 

Source:  Thesis  presented  at  the  School  of  Advanced  Airpower  Studies,  Air  University,  Maxwell 
Air  Force  Base,  Alabama. 

Year:  1996 
Abstract: 

This  is  a  study  of  the  applicability  of  prospect  theory  to  military  decision  making.  Prospect  theory 
posits  that  the  decision  maker's  reference  point  determines  the  domain  in  which  he  makes  a 
decision.  If  the  domain  is  one  of  losses,  the  decision  maker  will  tend  to  be  risk  seeking,  if  gains, 
then  he  will  be  risk  averse.  The  author  proposes  that  if  prospect  theory's  propositions  are  correct, 
then  it  may  be  possible  for  the  decision  maker,  by  assessing  his  own  domain,  to  make  better 
informed  decisions.  One  implication  of  this  study  is  that  if  he  can  do  the  same  for  a  subordinate  or 
for  an  enemy,  he  may  be  better  able  to  predict  their  responses  in  a  given  situation.  The  project's 
goal  is  to  develop  a  framework  for  assessing  risk  propensity.  It  does  this  by  first  describing  the 
military  decision  making  process  and  concluding  that  it  is  a  rational  decision  making  process.  Next, 
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the  paper  describes  prospect  theoiy  and  matches  the  key  aspects  of  the  theory  with  the  military 
decision  making  process.  Next,  it  proposes  a  framework  for  assessing  risk  propensity.  The  theory  is 
tested  in  a  case  study  of  General  Dwight  D.  Eisenhower's  1 944  decision  to  launch  Operation 
Market  Garden.  This  decision  is  analyzed  in  terms  of  Graham  Allison's  three  models  for  decision 
making  and  prospect  theory  to  determine  which  model  or  theory  seems  to  provide  the  best 
explanations  for  Eisenhower's  decision.  The  last  chapter  applies  the  risk  propensity  framework  to 
the  case  study  to  test  if  it  can  predict  risk  propensity  and  its  impact  on  decision  making.  The  author 
concludes  that  prospect  theory's  propositions  are  valid  and  that  this  theory  provides  a  prescriptive 
way  to  consider  decision  making  under  risk.  Although  prospect  theory  does  not  predict  the  choice  a 
decision  maker  will  select,  it  should  reveal  his  bias  toward  a  risky  or  over  cautious  solution. 

Notes: 

The  author  is  interested  in  understanding  how  military  commanders  make  decisions  about  risk. 
Prospect  theory  is  used,  and  this  theory  indicates  that  the  perspective  of  the  decision  maker 
influences  risk  decisions;  that  is,  is  the  decision  maker  thinking  about  possible  gains  or  losses? 

The  author  discusses  the  expected  utility  model,  which  assumes  that  preference  ordering  is 
constant,  whereas  actual  behaviour  contradicts  this  assumption.  Prospect  theory  allows  for 
differences  in  preferences  depending  on  the  loss  or  gain  focus  of  the  decision  maker.  An  example 
of  this  is  when  people  are  more  likely  to  engage  in  risky  behaviour  in  a  game  if  they  are  losing  than 
if  they  are  winning  (e.g.,  pulling  the  goalie  in  hockey). 

Outcomes  are  thought  of  as  changes  from  a  reference  point  -  the  reference  point  is  itself  changed 
by  how  the  decision  maker  evaluates  and  edits  the  decision  problem  -  editing  and  framing  the 
problem  are  dynamic  processes,  which  can  change  the  assessment  of  a  particular  outcome  from  a 
gain  to  a  loss,  and  vice  versa.  Thinking  of  an  outcome  as  a  gain  or  a  loss  will  influence  the 
assessment  of  risk,  which  will  in  turn  influence  decisions. 

Prospect  theory  does  not  allow  for  the  evaluation  of  a  decision  as  good  or  bad;  however,  it  does 
allow  a  decision  maker  to  assess  whether  they  are  likely  to  make  a  risk  averse  or  risk  seeking 
decision  by  determining  the  reference  point  and  domain  for  the  decision  which  are  based  on 
framing  effects.  This  can  be  used  in  a  metacognitive  fashion  to  mitigate  overly  risk  seeking  or  risk 
adverse  behaviours  and  possibly  mitigate  this  bias. 

Several  limitations  of  prospect  theory  are  noted,  including  the  possibility  of  individual  differences, 
influences  of  other  variables  such  as  uncertainty  can  influence  judgments,  and  the  fact  that 
reference  points  can  change  during  decision  making,  making  it  difficult  to  determine  whether  a  risk 
assessment  will  be  stable. 

Prospect  theory,  and  therefore  this  paper,  is  relevant  to  DDM  in  several  ways.  It  highlights  the  need 
to  think  of  decision  making  outside  of  the  rational  “box”,  but  demonstrates  in  a  relatively  concrete 
way  a  specific  method  for  applying  to  particular  situations.  How  risk  influences  decision  making 
can  be  thought  of  as  a  heuristic,  a  bias,  and  also  a  way  in  which  cues  are  integrated.  Investigating 
prospect  theory  has  the  potential  for  providing  ways  of  mitigating  the  tendency  to  inappropriately 
make  risky  and  risk  adverse  decisions. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  CUE 
INTEGRATION _ 

To  be  used?  INCLUDE  IF  RELEVANT 
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53.  Title:  Affect,  risk,  and  decision  making 

Author(s):  Slovic,  P.,  Peters,  E.,  Finucane,  M.  L.,  &  MacGregor,  D.  G. 

Source:  Health  Psychology,  24,  S35-S40. 

Year:  2005 _ _ 

Abstract: 

Risk  is  perceived  and  acted  on  in  2  fundamental  ways.  Risk  as  feelings  refers  to  individuals'  fast, 
instinctive,  and  intuitive  reactions  to  danger.  Risk  as  analysis  brings  logic,  reason,  and  scientific 
deliberation  to  bear  on  risk  management.  Reliance  on  risk  as  feelings  is  described  with  ""the  affect 
heuristic.""  The  authors  trace  the  development  of  this  heuristic  across  a  variety  of  research  paths. 
The  authors  also  discuss  some  of  the  important  practical  implications  resulting  from  ways  that  this 
heuristic  impacts  how  people  perceive  and  evaluate  risk,  and,  more  generally,  how  it  influences  all 
human  decision  making.  Finally,  some  important  implications  of  the  affect  heuristic  for 
communication  and  decision  making  pertaining  to  cancer  prevention  and  treatment  are  briefly 
discussed. 

Notes: 

The  authors  compare  and  contrast  the  assessment  of  risk  based  on  emotions  or  logic.  They  portray 
the  use  of  emotional  assessments  of  risk  as  the  “affect  heuristic”.  The  emotional  assessment  of  risk 
is  assumed  to  be  much  older  in  terms  of  human  evolution,  and  the  use  of  logic  a  relatively  recent 
means  of  “boosting”  risk  assessment.  Affective  risk  assessment  is  quick  and  occurs  outside  of 
consciousness,  and  seems  to  heavily  influence  further  information  processing  and  judgment. 

The  authors  note  the  finding  that  risk  and  benefit  tend  to  be  positively  correlated  in  the  world,  but 
people  perceive  them  as  negatively  correlated.  This  inverse  perceived  relationship  tends  to  be 
linked  to  the  strength  of  positive  or  negative  affect  associated  with  that  activity.  If  affect  is  positive 
toward  an  activity,  the  risk  is  judged  as  low  and  benefit  as  high;  the  opposite  occurs  if  affect  is 
negative  (risk  is  judged  as  high  and  benefit  as  low).  If  the  risk  or  benefit  is  manipulated  in 
someone’s  perception,  the  other  is  affected  as  well. 

The  authors  note  that  the  framing  of  information  also  influences  decisions  -  probabilities  tend  to  be 
assessed  as  less  probable  when  discussed  in  terms  of  percentages  rather  than  absolute  numbers 
(e.g.,  20%  is  judged  as  less  likely  than  20  out  of  100).  This  can  profoundly  affect  decision  making 
with  respect  to  probabilistic  judgment. 

This  article  is  relevant  to  the  use  of  heuristics  and  subsequent  biases,  both  in  risk  assessment  and 
other  decision  making  contexts.  There  is  no  explicit  link  to  DDM  contexts,  but  the  heuristic  would 
likely  be  at  play  in  any  type  of  situation. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES 
To  be  used?  INCLUDE  IF  RELEVANT 


54.  Title:  Validation  of  the  recognition-primed  decision  model  and  the  roles  of  common-sense 
strategies  in  an  adversarial  environment 

Author(s):  Soh,  B.  K. 

Source:  Dissertation  submitted  to  Virginia  Polytechnic  Institute  and  State  University,  Blacksburg, 
VA. 
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Year:  2007 _ 

Abstract: 

This  dissertation  set  out  to  understand  the  decision  processes  used  by  decision  makers  in 
adversarial  environment  by  setting  up  an  adversarial  decision  making  microworld,  as  an 
experimental  platform,  using  a  Real  Time  Strategy  (RTS)  game  called  Rise  of  Nations  (RON).  The 
specific  objectives  of  this  dissertation  were:  (1)  Contribute  to  the  validation  of  Recognition-Primed 
Decision  (RPD)  model  in  a  simulated  adversarial  environment;  (2)  Explore  the  roles  of  common- 
sense  strategies  in  decision  making  in  the  adversarial  environment;  and  (3)  Test  the  effectiveness  of 
training  recommendations  based  on  the  RPD  model.  Three  related  experimental  studies  were  setup 
to  investigate  each  of  the  objectives.  Study  1  found  that  RPD  model  was  partly  valid  where  RPD 
processes  were  prevalently  used  but  other  decision  processes  were  also  important  in  an  adversarial 
environment.  A  new  decision  model  (ConPAD  model)  was  proposed  to  capture  the  nature  of 
decision  making  in  the  adversarial  environment.  It  was  also  found  that  cognitive  abilities  might 
have  some  effects  on  the  types  of  decision  processes  used  by  the  decision  makers.  Study  2  found 
that  common-sense  strategies  were  prevalent  in  the  adversarial  environment  where  the  participants 
were  able  to  use  all  but  one  of  the  warfare  related  strategies  extracted  from  literature  without 
teaching  them.  The  strategy  familiarization  training  was  not  found  to  significantly  improve 
decision  making  but  showed  that  common-sense  strategies  were  prevalent  and  simple 
familiarization  training  was  not  sufficient  to  produce  differences  in  strategy  usage  and 
performances  from  the  novice  participants.  Study  3  also  found  that  RPD  based  training  (cue- 
recognition  and  decision  skill  training)  were  not  significant  in  producing  better  performance 
although  subjective  feedback  found  such  training  to  be  useful.  However,  the  participants  with  RPD 
based  training  conditions  were  able  to  perform  on  the  same  level  as  the  expert  participants  bridging 
the  gap  between  novices  and  experts.  Based  on  the  findings,  it  was  recommended  that  decision 
training  should  involve  not  just  RPD  based  training,  but  comparisons  of  attributes  as  well.  A  more 
interactive  training  combining  common-sense  strategies,  cue-recognition  and  decision  skill  training 
might  be  more  useful.  More  theoretical  experimentation  would  be  required  to  validate  the  new 
decision  model  proposed  in  this  dissertation. 

Notes: 

RPD  is  characteristic  of  experts,  and  involves  choosing  one  most  likely  alternative  and  COA.  In 
contrast,  novices  tend  to  concurrently  assess  several  options  and  not  necessarily  know  which 
critical  factors  are  most  useful  in  making  decisions.  The  author  also  notes  that  individual 
differences  in  attention  and  working  memory  relates  to  the  RPD  model. 

The  article  relates  schemas  to  RPD  -  schemas  are  how  expert  knowledge  may  be  structured. 
Strategies  are  considered  to  be  the  sequence  of  actions  taken  when  a  schema  is  instantiated,  and 
these  strategies  may  be  thought  of  as  heuristics. 

“Commonsense  strategies”  are  described  as  heuristics  which  are  extended  to  day-to-day  reasoning 
and  decision  making;  another  way  of  describing  them  is  as  rules  of  thumb  in  daily  reasoning  (e.g., 
“divide  and  conquer”).  What  makes  a  good  decision  maker  is  the  ability  to  identify  the  right 
strategy  for  the  situation. 

Discusses  the  importance  of  experience  for  pattern  recognition,  and  the  practical  reasons  for  using 
simulation  (compressed  time  frame).  Discusses  training  for  decision  making  in  natural  tasks  (p.  25) 
from  previous  work,  and  variables  which  should  be  considered  when  choosing  training  methods 
(e.g.,  workload,  time  stress).  Goals  for  training  should  also  be  considered,  including  recognizing 
patterns,  making  perceptual  discriminations,  detecting  anomalies  (p.  30).  Note  that  research 
validating  these  methods  was  only  subjective  in  nature,  as  it  was  considered  impossible  to  actually 
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judge  the  appropriateness  of  decisions. 

The  author  of  this  dissertation  chose  a  laboratory  approach  to  provide  greater  control  over  the  task 
and  participants,  and  (presumably)  to  allow  more  empirical  evaluation  of  decisions  made. 
Participants  were  gathered  from  the  community,  and  people  with  real-time  strategy  game 
experience  were  considered  experts,  rather  than  people  with  experience  with  military  decision 
making. 

Evidence  suggests  that  people  use  multiple  decision-making  strategies,  and  experts  often  don’t  use 
RPD.  There  was  also  not  a  straightforward  link  between  cognitive  abilities  and  decision  processes. 
It  is  an  open  question  whether  any  participants  actually  reached  the  status  of  experts  in  the  game, 
which  could  significantly  affect  the  validity  of  the  results. 

Training  participants  in  commonsense  strategies  did  not  improve  performance.  The  actual 
strategies  used  tended  to  differ  between  groups,  but  there  were  a  large  number  of  strategies 
considered  and  family-wise  error  was  not  controlled  for.  One  trend  was  that  people  familiarized 
with  different  strategies  used  more  strategies;  making  people  aware  of  strategies  may  provide  some 
benefit  as  participants  may  see  more  options.  It  is  also  possible  that  large  effects  were  not  observed 
because  the  “common  sense”  strategies  were  just  that;  people  were  already  aware  of  them  and  used 
them,  and  were  at  ceiling  for  strategy  knowledge. 

Two  options  for  using  the  RPD  model  for  training  is  to  either  train  cue  recognition  or  train 
decision-makers  to  critique  their  decisions  and  improve  their  decision  making.  No  significant 
differences  in  level  or  type  of  training  was  found. 

This  dissertation  provides  an  interesting  discussion  of  types  of  training  that  could  be  useful  in 
developing  training  to  help  decision  makers,  but  evidence  was  inconclusive  about  which  methods 
would  be  most  useful.  Understanding  how  “common  sense”  reasoning  is  applied  in  new  domains  is 
a  useful  one,  but  it  is  uncertain  whether  this  is  used  in  addition  to  heuristics  developed  by  experts 
or  more  often  by  novices  who  have  not  created  expert  schemas. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES,  AND  PATTERN 
RECOGNITION 

To  be  used?  INCLUDE  IF  RELEVANT 


55.  Title:  Misperceptions  of  Feedback  in  Dynamic  Decision  Making 
Author(s):  Sterman,  J.  D. 

Source:  Organizational  Behavior  and  Human  Decision  Processes,  43, 301-335. 

Year:  1989 _ 

Abstract: 

In  recent  years  laboratory  experiments  have  shed  significant  light  on  the  behavior  of  economic 
agents  in  a  variety  of  microeconomic  and  decision-theoretic  contexts  such  as  auction  markets, 
portfolio  choice,  and  preference  elicitation.  Despite  the  success  of  experimental  techniques  in  the 
micro  domain,  there  has  been  relatively  little  work  linking  the  behavior  of  decision  makers  to  the 
dynamics  of  larger  organizations  such  as  corporations,  industries,  or  the  macroeconomy.  This  paper 
presents  a  laboratory  experiment  in  which  subjects  manage  a  simulated  economy.  Subjects  must 
invest  sufficient  capital  plant  and  equipment  to  satisfy  demand.  Subjects  were  given  complete  and 
perfect  information  regarding  the  structure  of  the  simulated  economy,  the  values  of  all  variables. 
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and  the  past  history  of  the  system.  Nevertheless,  the  overwhelming  majority  of  the  subjects 
generate  significant  and  costly  oscillations.  A  simple  decision  rule  based  on  the  anchoring  and 
adjustment  heuristic  is  shown  to  simulate  the  subjects'  decisions  quite  well.  Several  distinct  sources 
of  the  subjects'  poor  performance  are  identified  and  termed  “misperceptions  of  feedback.”  The 
decision  rule  is  related  to  various  models  of  economic  fluctuations;  implications  for  experimental 
investigation  of  dynamic  decision  making  in  aggregate  systems  are  explored. 

Notes: 

The  author  conducted  a  study  that  ran  participants  through  a  DDM  experiment  and  then  modeled 
their  responses  to  determine  what  (if  any)  heuristics  they  seemed  to  be  following.  The  system 
chosen  for  the  experiment  was  called  the  multiplier-accelerator  model  of  capital  investment. 
Participants  in  this  study  included  both  students  and  economic  professionals  knowledgeable  about 
economic  and  control  theory.  Despite  only  a  single  simple  system  input  change,  only  8%  of  the 
participants  were  able  to  re-establish  equilibrium  before  the  end  of  the  simulation.  Follow-on 
articles  about  this  study  have  suggested  the  anchoring  and  adjustment  heuristic  is  a  good 
description  of  the  rule  modeled  by  the  subjects. 

One  implication  of  the  suggested  response  rule  is  a  lack  of  consideration  of  feedback.  Removing 
the  multiplier  feedback  from  the  system,  the  response  rule  becomes  locally  rational.  Thus,  a 
number  of  specific  causal  factors  that  appear  to  be  ignored  by  the  subjects  are  discussed 
(misperception  of  time  delays,  misperceptions  of  feedbacks  from  decisions  to  the  environment  and 
a  tendency  to  think  in  causal  single  strand  series).  Arguably,  this  shows  a  lack  of  pattern 
recognition  on  the  part  of  the  subjects. 

No  where  in  this  study  is  there  an  exploration  of  the  cues/information  actually  taken  into  account 
by  the  subjects  or  the  salience  of  the  different  information  categories.  Thus,  it  would  be  interesting 
to  see  what  (if  any)  effects  a  different,  perhaps  optimized,  interface  would  have  on  the  results. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  CUE  INTEGRATION,  PATTERN 
RECOGNITION,  HEURISTICS,  and  BIASES. _ 

To  be  used?  INCLUDE  IF  RELEVANT 


56.  Title:  My  Intuition  Works,  so  Why  Mess  with  this  Decision  Analysis  Stuff 
Author(s):  Stonebraker,  J.  S. 

Source:  Report  prepared  for  the  Department  of  Mathematical  Sciences,  US  Air  Force  Academy. 
Year:  1990 _ 

Abstract: 

This  report  describes  possible  shortcomings  of  unaided  decision  making  and  benefits  of  decision 
analysis.  The  premise  of  this  report  is  that  unaided  decision  making  only  works  for  simple 
decisions,  but  rarely  suffices  for  complicated  decision  problems.  Unaided  decision  making  is  a 
holistic  approach  using  the  decision  maker's  intuition  and  heuristics.  Unaided  decision  making  can 
result  in  biases  and  systematic  errors.  Decision  analysis  is  a  systematic  process  with  the  objective 
to  provide  insight  and  aid  a  decision  maker  in  making  better  decisions  by  being  consistent  and 
logical.  Decision  analysis  is  a  modelling  process  that  quantifies  uncertain  information  using 
probability  theory  and  risk  using  utility  theory.  Decision  analysis  involves  decomposing  the 
decision  problem  into  manageable  pieces  and  then  recombining  the  information  as  the  expectation 
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of  probabilities  and  utilities. 

Notes: 

This  report  staunchly  supports  rational  decision  making  and  is  critical  of  decision  makers  who 
follow  their  intuition.  Referring  to  intuition  as  “holistic”,  common  sense  and  heuristic  decision 
making,  the  author  prefers  to  decompose  problems  into  smaller  parts  and  analyse  those 
subcomponents  instead  -  calling  this  process  Decision  Analysis  (DA).  The  four  steps  to  this 
process  are:  formulate,  evaluate,  refine  and  decide. 

Citing  many  studies  supporting  the  sub-optimal  decision  making  that  humans  perform,  little 
recommendation  is  given  for  decision  making  under  scenarios  with  ambiguous  and  uncertain 
information.  Further,  though  time  is  listed  as  a  disadvantage  of  DA,  optimal  decision  strategy 
solutions  during  time  sensitive  scenarios  are  not  discussed. 

In  general,  the  author  is  presenting  a  perspective  with  little  supporting  evidence  against  intuition. 
This  viewpoint  seems  out  of  date  given  more  recent  research  (e.g.  Klein). 

CANNOT  BE  USED  TO  SUPPORT  INVESTIGATING  ANY  OF  THE  RELVANT  TOPICS. 
To  be  used?  DO  NOT  INCLUDE 


57.  Title:  The  Decision  Dilemma-  Cognitive  Bias 
Author(s):  Stormer,  W.  P. 

Source:  Paper  presented  to  the  U.S.  Army  War  College,  Carlisle  Barracks,  PA. 

Year:  1991 
Abstract: 

Military  decision-making  at  the  national  level  is  a  formidable  task.  The  environment  is  ambiguous 
and  constantly  changing.  Intelligence  systems,  as  good  as  they  are,  will  never  provide  perfect 
information.  The  stakes  involved  range  from  the  loss  of  thousands  of  lives  to  the  sovereignty  of  the 
nation.  Faced  with  this  difficult  task,  the  military  decision-maker  often  has  to  rely  on  intuitive 
processes  to  arrive  at  a  final  decision.  One  of  the  magnificent  qualities  of  human  beings  is  the 
ability  to  integrate  a  vast  array  of  factors  weighing  on  a  situation  and  come  up  with  a  viable 
solution.  This  process  involves  a  great  degree  of  intuitive  judgment.  While  there  are  many 
examples  of  great  military  decisions  made  substantially  on  the  basis  of  intuition,  there  is  also  a 
down  side  to  this  phenomenon.  Research  has  shown  that  there  are  cognitive  biases  which  can 
adversely  affect  the  decision-making  process.  This  paper  provides  an  analysis  of  these  biases  and 
offers  practical  suggestions  on  how  to  reduce  their  effects. 

Notes: 

Some  points  made  in  this  document  include:  The  environment  in  which  military  decisions  need  to 
be  made  is  very  dynamic;  Assessing  different  aspects  of  a  situation/  environment  are  necessarily 
subjective;  Decisions  are  often  made  using  intuitive  processes;  Cognitive  biases  inevitably  come 
into  play  within  such  a  decision-making  process;  Decision  makers  are  often  unaware  of  biases  and 
their  effects  on  the  decision  process. 

The  goal  of  the  report  is  to  provide  military  decision  makers  fundamental  awareness  of  cognitive 
biases  in  the  efforts  to  improve  decision-making  effectiveness. 

Decision  makers  operate  within  “bounded  rationality”  limited  by  time  constraints,  available  funds. 
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imperfect  information,  memory  limitations,  etc.  People  also  “satisfice”  in  decision-making 
processes,  meaning  that  they  seek  a  satisfactory  solution  rather  than  an  optimum  solution. 

The  author  suggests  that  understanding  the  biases,  their  root  causes,  and  learning  ways  to  reduce 
their  influence  will  improve  decision  making,  and  thus  this  is  the  purpose  of  the  report.  Examples 
of  biases  are  presented  and  it  is  argued  that  they  are  a  result  of  using  heuristics.  The  author  argues 
just  being  aware  of  cognitive  biases  can  help  avoid  them.  Improvements  in  the  quality  of  decision 
making  can  be  made  thorough  studying  the  decision-making  process.  Studying  previous  events  in 
which  biases  are  prevalent  may  provide  good  examples  that  help  people  to  avoid  biases  in  their 
own  decision  making. 

Similar  to  Jansen  (2007),  the  author  believes  that  simply  identifying  biases  and  making  them 
explicit  can  change  their  influence  in  decision  making.  This  is  not  necessarily  in  agreement  with 
more  resent  research. 

The  author  identifies  intuitive  decision  making  processes  as  helpful  but  impacted  by  biases  due  to 
the  use  of  heuristics  in  intuitive  processing. 

The  author  suggests  ways  of  improving  the  negative  impact  of  biases: 
awareness  of  biases  can  help  avoid  them 

increase  the  amount  of  time  spent  studying  and  understanding  the  decision  making  process 
at  a  training  level  (in  school). 

Using  historical  examples  to  understand  biases.  Knowing  detailed  accounts  of  history 
when  negative  outcomes  occur  can  help  to  understand  and  qualify  the  importance  of  good 
decision  making. 

The  decision-maker  should  take  time  to  reflect  on  the  courses  of  action  being  considered 
and  determine  if  there  are  dominant  themes. 

Decisions  should  consider  all  alternatives  as  much  as  possible. 

These  suggestions  would  not  fare  well  in  a  DDM  context  where  time  constraints  and  cognitive 
workload  are  severely  limited. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS,  BIASES _ 

To  be  used?  FOR  CONSIDERATION 


58.  Title:  Pattern  recognition  training  for  combat  leaders:  Sample  training  package 
Author(s):  Thronesbery,  C.,  Sullivan,  B.  D.,  &  Rhoads,  R. 

Source:  Report  prepared  for  the  United  States  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  ARI  Report  No.  96-17. 

Year:  1996 
Abstract: 

Lesson  plans  were  developed  to  facilitate  and  increase  pattern  recognition  for  combat  leaders. 
Following  an  overview  of  the  pattern  recognition  approach  to  training,  this  report  contains  a 
demonstration  package  that  illustrates  implementation  of  the  pattern  recognition  approach  to 
training.  An  extensive  set  of  storyboards  offers  instructional  material  covering  enemy  situation 
understanding  from  an  elementary  to  an  advanced  level.  It  incorporates  the  development  of  key 
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pattern  concepts  that  are  used  in  threat  evaluation  during  Intelligence  Preparation  of  the  Battlefield. 
The  demonstration  materials  additionally  provide  training  for  Soviet  tactical  considerations  of  a 
motorized  rifle  regiment  during  two  tactical  scenarios. 

Notes: 

This  article  contains  a  brief  description  of  the  goals  and  contents  of  a  training  package  to  support 
pattern  recognition;  the  majority  of  the  report  contains  storyboards  to  be  used  for  training. 

The  goal  is  to  show  development  of  key  concepts  (patterns)  necessary  for  threat  evaluation  during 
intel  preparation  of  the  battlefield. 

The  authors  stress  the  importance  of  overlearning  in  training  (i.e.,  training  to  the  point  of 
automaticity),  as  well  as  the  importance  of  hierarchical  training  (teaching  basic  skills  and  then 
more  advance  skills)  in  pattern  recognition.  Also  stressed  is  the  importance  of  mental  models  in 
pattern  recognition;  understanding  the  deeper  principles  which  make  patterns  similar. 

Types  of  training  exercises  include  identifying  stimuli  which  are  similar  to  a  prototype,  identifying 
variations  of  a  prototype  (including  which  components  might  be  missing),  and  assigning  examples 
to  prototypes.  Feedback  is  provided,  including  information  about  whether  the  response  was  correct 
or  incorrect,  and  reason  for  error  when  it  occurred  (i.e.,  why  an  error  is  an  error). 

Although  this  article  contains  specific  examples  of  training  materials  for  pattern  recognition,  it 
contains  only  minimal  information  about  the  justification  for  choices  made  and  strategies  for 
training.  More  detail  is  needed  to  understand  the  rationale  for  decisions  and  how  to  understand  the 
applicability  of  the  methods  detailed  here. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  PATTERN  RECOGNITION _ 

To  be  used?  FOR  CONSIDERATION-  probably  need  to  find  the  justification  research  rather  than 
using  this  reference 


59.  Title:  Precis  of  Simple  heuristics  that  make  us  smart 
Author(s):  Todd,  P.  M.  &  Gigerenzer,  G. 

Source:  Behavioral  and  Brain  Sciences,  23,  727-741. 

Year:  1999 
Abstract: 

How  can  anyone  be  rational  in  a  world  where  knowledge  is  limited,  time  is  pressing,  and  deep 
thought  is  often  an  unattainable  luxury?  Traditional  models  of  unbounded  rationality  and 
optimization  in  cognitive  science,  economics,  and  animal  behavior  have  tended  to  view  decision¬ 
makers  as  possessing  supernatural  powers  of  reason,  limitless  knowledge,  and  endless  time.  But 
understanding  decisions  in  the  real  world  requires  a  more  psychologically  plausible  notion  of 
bounded  rationality.  In  Simple  heuristics  that  make  us  smart,  we  explore  fast  and  frugal 
heuristics — simple  rules  in  the  mind’s  adaptive  toolbox  for  making  decisions  with  realistic  mental 
resources.  These  heuristics  can  enable  both  living  organisms  and  artificial  systems  to  make  smart 
choices  quickly  and  with  a  minimum  of  information  by  exploiting  the  way  that  information  is 
structured  in  particular  environments.  In  this  precis,  we  show  how  simple  building  blocks  that 
control  information  search,  stop  search,  and  make  decisions  can  be  put  together  to  form  classes  of 
heuristics,  including:  ignorance-based  and  one-reason  decision  making  for  choice,  elimination 
models  for  categorization,  and  satisficing  heuristics  for  sequential  search.  These  simple  heuristics 
Cue  integration  in  Dynamic  Decision  making 


Page  136 


Humansyj’/e/nj' 


HUMAN  SYSTEMS 


perform  comparably  to  more  complex  algorithms,  particularly  when  generalizing  to  new  data — that 
is,  simplicity  leads  to  robustness.  We  present  evidence  regarding  when  people  use  simple  heuristics 
and  describe  the  challenges  to  be  addressed  by  this  research  program. 

Notes: 

The  first  chapter  of  this  book  was  read. 

The  authors  discuss  the  use  of  heuristics  in  decision  making.  They  investigate  different  types  of 
heuristics,  performance  when  using  heuristics,  and  methodologies  to  investigate  and  study 
heuristics. 

The  challenges  of  decision  making  are  due  to  the  uncertainty  and  time  limitations  within  the 
decision-making  environment. 

They  discuss  the  differences  between  unbounded  rationality,  described  as  knowing  everything 
about  the  situation  that  is  required  (as  God  might);  optimization  under  constraints,  describing  how 
our  decisions  are  grounded  in  uncertainty;  and  bounded  rationality,  described  as  “satisficing”, 
which  is  when  there  may  be  no  optimal  method  for  making  a  decision,  thus  a  few  options  are 
explored  and  one  is  chosen  often  as  a  result  the  structure  of  the  information  determines  much  of  our 
behaviour.  Bounded  rationalit,  often  focuses  on  the  use  of  heuristics  such  as  “one-reason  decision 
making”. 

The  authors  then  discuss  heuristic  principles  for  guiding  research  and  decision  making.  How 
heuristics  may  be  used  in  real  world  environments  is  also  discussed,  as  is  the  use  of  “fast  and 
frugal”  heuristics.  Summaries  and  introductions  to  different  chapters  of  the  book  focus  on  these 
topics. 

The  environments  explored  are  all  related  to  simple  decision  making  tasks.  The  authors  discuss  the 
challenges  within  decision  making  (DM)  are  based  on  the  uncertainty  of  situational  and  temporal 
factors. 

Working  memory  capacities  are  one  limitation  thus  is  the  reason  heuristics  are  used.  This  helps  to 
limit  the  analytic  processing  that  people  have  to  do. 

The  environment  and  structure  of  the  information  largely  determines  how  we  behave.  This  may 
mean  that  much  of  our  behaviour  within  these  constrained  environments  is  beyond  our  control. 
Coping  strategies  in  dealing  with  such  situations  are  identified  as  heuristics.  Ideas  of  using 
heuristics  is  to  avoid  and  compensate  for  the  environmental  challenges 

Our  cognitive  capacities  and  abilities  are  limited  because  of  the  lack  of  information. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING:  HEURISTICS _ 

To  be  used?  FOR  CONSIDERATION 

Particular  chapters  of  this  book  may  be  useful  if  relevant 


60.  Title:  Situation  Assessment  and  Hypothesis  Testing  in  an  Evolving  Situation 
Author(s):  Tolcott,  M.  A.,  Marvin,  F.  F.,  &  Bresnick,  T.  A. 

Source:  Report  prepared  for  the  United  States  Army  Research  Institute  for  the  Behavioural  and 
Social  Sciences,  ARI  Report  No.  96-34. 

Year:  1996 
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Abstract: 

This  research  investigated  the  effects  of  early  judgment  on  (1)  the  handling  of  new  information, 
some  of  which  confirmed  and  some  of  which  contradicted  the  early  judgment,  and  (2)  the  selection 
of  hypothesis-testing  indicators.  The  context  was  situation  assessment  by  Army  intelligence 
analysts  during  an  evolving  battlefield  scenario.  Unaided  analysts  typically  ignored  or 
underweighted  contradictory  evidence;  their  confidence  in  their  early  judgment  tended  to  rise.  A 
second  group  was  given  a  brief  tutorial  on  common  decision  biases  and  graphic  displays  that 
fostered  awareness  of  uncertainty;  in  this  group  the  tendencies  were  reduced  (but  not  eliminated), 
and  one-half  of  the  group  reversed  their  judgment  at  least  once.  A  third  group  selected  indicators; 
however,  in  the  face  of  balanced  feedback,  their  confidence  remained  constant  rather  than  rising. 
The  findings  support  the  extension  of  confirmation  bias  theories  to  trained  personnel  performing 
realistic  tasks.  In  addition,  the  results  suggest  that  when  decision  makes  the  indicators  they  believe 
to  be  important,  they  pay  more  attention  to  contradictory  evidence  than  when  they  are  the  passive 
recipients  of  new  information. 

Motes: 

Focused  on  confirmation  and  overconfidence  bias,  this  report  summarizes  two  previous  studies  and 
details  a  third  study  involving  intelligence  analysts  (IA).  Findings  suggest  that  confirmation  and 
overconfidence  bias  occurs  in  natural  settings,  as  the  IAs  taking  a  part  in  the  study  repeatedly 
focused  on  areas  they  believed  the  enemy  would  attack  and  were  overconfident  in  their 
assessments.  The  authors  suggest  a  number  of  cognitive  skills  that  should  be  addressed. 

This  report  was  very  focused  on  these  two  specific  biases,  and  provides  little  in  the  way  of  training 
suggestions  for  DDM. 

CAN  BE  USED  TO  SUPPORT  INVESTIGATING  BIASES. 

To  be  used?  DO  NOT  INCLUDE 
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Annex  B 


This  section  contains  a  summary  of  each  of  the  four  areas  reviewed.  Below  are  definitions  that  we 
used  as  guidelines  in  our  review.  Note  that  there  are  many  different  definitions  in  the  literature  and 
so  these  definitions  are  not  to  be  taken  as  consistently  held  by  other  authors. 

■  Cue  integration  is  the  process  by  which  information  is  selected  from  the  environment, 
weighted,  and  combined  to  serve  as  the  input  to  a  decision. 

■  Pattern  Recognition  is  the  process  of  applying  knowledge  about  the  relationships  between 
aspects  of  the  environment  (e.g.,  patterns).  Pattern  recognition  is  usually  thought  to  be  a 
function  of  the  similarity  between  the  state  of  the  environment  and  the  patterns  stored  in 
memory,  which  are  usually  learned  through  experience. 

■  Heuristics  are  shortcuts  in  decision  making,  generally  designed  to  find  a  decision  that  is 
“good  enough”  (i.e.,  satisficing).  Heuristics  are  often  contrasted  with  analytical  methods 
which  systematically  examine  all  possibilities  and  are  intended  to  arrive  at  an  optimal 
decision  (definition  adapted  from  Adams,  Rehak,  Brown,  &  Hall,  2009). 

■  Biases  are  systematic  (i.e.,  non-random)  errors  in  decision  making  (Adams  et  al.,  2009). 

Cue  integration 

Drawing  from  our  understanding  of  the  term  “cue  integration”,  a  number  of  general  themes, 
challenges  and  training  related  issues  can  be  pulled  from  the  literature.  These  are  discussed  below. 

General  Themes 

Cue  integration  involves  the  use  of  cues,  hints  or  data  in  decision  making.  These  cues  are  used  to 
determine  which  decision,  action  or  judgement  is  most  appropriate.  In  any  given  situation  a 
decision  maker  must  learn  what  cues  to  pay  attention  to,  how  valid  those  cues  are  and  the  how  to 
integrate  the  cues  together.  One  issue  not  discussed  in  detail  in  the  literature  is  determining  what  a 
cue  is.  For  our  purposes,  a  cue  can  be  considered  any  piece  of  data  used  in  decision  making.  This 
includes  data  pertaining  to  the  task  at  hand  (e.g.,  the  location  of  enemy  groups),  data  internal  to  the 
decision  maker  (e.g.,  affect),  and  data  from  the  environment  (e.g.,  time  constraints).  Each  of  these 
categories  of  cues  affects  the  decision  maker  differently,  potentially  changing  the  resulting  decision 
(Kerstholt  &  Willems,  1993;  Schroeder,  2005;  Famewo  et  al.,  2005). 

Arguably,  cues  -  especially  task  cues  -  must  be  perceived  before  they  can  be  incorporated  into  the 
decision  making  process.  The  process  of  perception  is  unavoidable;  however,  cue  perception  is  not 
a  straightforward  recording  of  reality.  Instead,  our  perceptions  are  believed  to  be  constructed 
(Jones,  2005),  and  that  construction  process  is  affected  by  internal  cues  and  environmental  cues. 
This  makes  determining  what  cues  are  being  used,  and  whether  or  not  those  cues  are  being  used  in 
a  decision  correctly  and/or  effectively  a  challenge.  This  is  especially  the  case  in  DDM  scenarios 
where  the  quantity  of  task  cues  is  large,  and  the  interrelationship  between  cues  can  be  complicated 
and  changes  over  time. 

Taking  a  closer  look  at  the  actual  cue  integration  process,  a  number  of  cognitive  strategies  and 
procedures  are  discussed  in  the  literature  concerning  how  cues  are  integrated.  Hammond  (1988) 
outlines  two  distinct  cognitive  strategies  based  on  the  way  information  is  presented:  1)  Pattern 
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seeking:  induced  by  displays  of  information  of  a  high  degree  of  perceptual  or  conceptual 
organization;  and  2)  Functional-relation  seeking:  induced  by  displays  that  demand  description  and 
prediction  on  the  variables.  Cue  integration  logically  occurs  in  both  of  these  strategies,  though 
different  cue  integration  strategies  would  be  applied. 

There  have  been  some  studies  suggesting  that  the  cognitive  processes  involved  in  cue  integration 
vary  based  on  the  type  of  output  required.  For  example,  Juslin,  Olsson  and  Olsson  (2003)  suggest 
that  different  cognitive  processes  are  used  to  determine  categorization  (i.e.  allocating  between  2  or 
more  categories)  versus  multiple  cue  judgements  of  a  continuous  variable  (e.g.,  probability  of 
precipitation  given  weather  information). 

Finally,  the  use  of  cues  is  heavily  discussed  in  comparisons  of  compensatory  and  non-compensatory 
decision  strategies.  Compensatory  strategies  involve  integrating  all  information  together  to  be  used  in 
the  decision,  whereas  non-compensatory  strategies  involve  disregarding  certain  information  (e.g.,  in 
the  “choose  the  best”  heuristic  only  the  most  valid  cue  is  considered)  (Pachur  &  Hertwig,  2006).  Non¬ 
compensatory  cue  usage  strategies  seem  to  be  prevalent  in  dynamic  tasks,  which  are  reasonable  given 
that  non-compensatory  strategies  often  yield  very  good  results  compared  to  the  more  cognitively 
demanding  compensatory  strategies  (Rothrock  &  Kirlik,  2003). 

While  each  of  these  examples  could  be  considered  micro-decisions  in  the  broader  DDM  process, 
the  evidence  for  different  cognitive  processes  implies  that  cues  are  not  only  used  to  make 
decisions,  but  also  used  to  determine  which  strategy  to  use  to  make  the  decision  (Gonzales,  Lerch, 

&  Lebiere,  2003)  as  different  cues  are  used  to  determine  which  cognitive  process  to  use.  This  could 
be  especially  applicable  to  DDM  training. 

Cue  Integration-based  challenges 

The  effects  of  information  presentation  (e.g.,  framing)  of  cues  on  decisions  have  been  extensively 
researched.  The  order,  format,  salience,  context,  affect,  etc.  of  cues  impact  how  and  if  a  cue  is 
incorporated  into  decision  making  (Adelman  &  Bresnick,  1995;  Bois,  2002;  Dunwoody, 

Haarbauer,  Mahan,  Marino,  &  Tang,  2000;  Famewo  et  al.,  2005;  Jones,  2005;  Payne,  Bettman,  & 
Johnson,  1 992;  Tolcott,  Marvin,  &  Bresnick,  1 996).  Some  of  these  studies  conclude  that  the  effects 
occur  outside  of  consciousness  (e.g.,  Slovic  et  al.,  2005)  while  other  effects  are  more  accessible 
(Schulz,  1 996).  Collectively,  these  effects  lead  to:  neglecting  potentially  relevant  problem 
information,  ignoring  values  when  they  are  difficult  to  estimate,  inaccurately  weighting  cue 
validity,  and  the  “general  tendency  to  accept  problems  as  given  and  the  segregation  of  the  decision 
problem  at  hand  from  its  broader  context”  (p.  116,  Payne,  Bettman,  &  Johnson,  1992). 

Defining  “cue  validity”  as  “the  probability  that  a  cue  will  lead  to  the  correct  decision  given  that  it 
discriminates  between  the  alternatives”  (p.  1399),  Garcia-Retamero,  Wallin,  and  Dieckmann 
(2007)  looked  at  how  decision  makers  select  cues,  and  how  estimations  of  cue  validity  is 
performed.  Cues  can  be  present  or  not  present,  but  they  can  also  be  associated  with  uncertainty.  A 
cue  may  be  ambiguous,  inaccurate,  or  probabilistic.  Thus,  determining  cue  validity  can  also  be  seen 
as  a  cognitive  challenge. 

Additional  challenges  alluded  to  above  include  the  sheer  quantity  of  cues  which  can  exceed  the 
cognitive  limitations  of  the  decision  maker  (Adelman  &  Bresnick,  1995;  Jones,  2005),  and  the  use 
of  cues  to  select  decision  strategies.  The  limitations  of  working  memory  restrict  the  ability  to 
process,  and  integrate  multiple  sources  of  information  resulting  in  poor  decision  making 
performance  (Famewo,  Matthews,  &  Lamoureux,  2007;  Gibb,  2000;  Jacobs  &  Graver,  1998). 

These  restrictions  also  impact  the  ability  to  identify  relationships  between  cues  (Garcia-Retamero, 
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Wallin,  &  Dieckman,  2007).  Jacobs  and  Graver  (1998)  argue  that  training  and  experience  can 
lessen  the  effects  of  limited  information  processing  and  memory  capabilities. 

Training  Cue  integration  in  a  DDM  context 

Providing  subjects  with  perfect  information  does  not  mean  that  subjects  understand  or  perceive  all  the 
cues.  This  is  the  case  for  general  data  as  well  as  feedback.  Logically,  feedback  could  be  considered  a 
cue  in  DDM,  or  information  about  cue  validity  given  the  cyclical  and  continuous  nature  of  DDM 
scenarios.  However,  Sterman  (1989)  found  that  in  DDM  scenarios,  subjects  show  a  general  lack  of 
consideration  of  feedback,  including  misperceptions  of  time  delays,  misperceptions  of  feedbacks  from 
decision  to  the  environment  and  a  tendency  to  think  in  causal  single  strand  series.  Further,  Gonzales, 
Lerch,  and  Lebiere  (2003)  found  that  when  modelling  behaviour,  models  without  feedback  were  more 
accurate  at  representing  human  performance  that  those  with  feedback.  Collectively,  these  studies 
suggest  that  feedback  is  not  readily  incorporated  as  a  cue  in  dynamic  decision  making. 

If  feedback  is  not  used,  how  do  decision  makers  learn  appropriate  relations  between  cues?  Juslin, 
Olsson,  and  Olsson  (2003)  suggest  that  exemplar  memory  is  used  as  a  guide  to  determine  which 
cues  should  be  selected.  Other  studies  have  tried  to  impress  rules  upon  participants  in  order  to 
increase  the  usage  of  appropriate  cues.  Bois  (2002)  found  that  if  you  provide  participants  with  a 
decision  rule  that  focuses  their  attention  on  proper  cues  and  how  to  weigh  their  importance,  subject 
performance  was  improved.  Costello  (1992)  further  found  that  subject  performance  in  a  DDM 
environment  was  better  when  the  easier  of  two  projects  was  managed  first  -  showing  that  training 
with  increased  difficulty  could  be  effective  in  DDM  environments. 

Further  (as  alluded  to  above),  training  on  the  use  of  cues  to  select  a  decision  strategy  could  be 
effective.  Butterfield  (1993)  found  with  intelligence  officers  that  efforts  focusing  on  the 
adaptability  of  concepts  and  how  to  apply  critical  judgements  were  a  more  effective  training 
approach  than  explicitly  training  participants  to  increase  their  objectivity. 

In  general,  DDM  decision  makers  should  be  trained  in  a  way  that  increases  robustness  (i.e.  in 
strategies  that  are  suitable  across  a  wide  variety  of  options  due  to  variability  in  the  way  the  system 
could  evolve)  rather  than  for  optimality  given  the  current  or  predicted  situation  (Busemeyer  & 
Pleskak,  2008). 

Pattern  Recognition 

Drawing  from  our  understanding  of  the  term  “pattern  recognition”,  a  number  of  general  themes, 
challenges  and  training  related  issues  can  be  pulled  from  the  literature.  Pattern  Recognition  is  the 
process  of  applying  knowledge  about  the  relationships  between  aspects  of  the  environment  (e.g., 
patterns)  and  knowledge  of  what  decisions  are  appropriate  and  what  the  effects  of  those  decisions 
are  likely  to  be.  It  is  a  complicated  process  that  involves  selecting,  combining,  categorizing,  and 
understanding  information.  Interest  in  this  area  of  decision  making  has  grown  with  recent  interest 
in  intuitive  and  naturalistic  decision  making,  including  Recognition  Primed  Decision  Making 
(RPDM;  Klein  and  Crandall,  1 996).  These  approaches  often  treat  pattern  recognition  as  a  “black 
box”,  whose  input  is  data  and  whose  output  is  categorization  and/or  a  deeper  understanding. 

However,  a  closer  look  at  pattern  recognition  is  warranted. 
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General  Themes 

Pattern  recognition  can  take  place  at  either  a  conscious  (e.g.,  deliberately  manipulating  mental 
models)  or  subconscious  (e.g.,  affective  response)  level.  Logically,  pattern  recognition  is  a  skill 
that  increases  with  familiarity  and  experience.  Experts  who  are  effective  at  pattern  recognition  are 
able  to  also  recognize  novel  situations  and  exceptions  (Cohen  et  al.,  1999).  As  decision  makers 
become  more  experienced  -  and  presumably  better  at  pattern  recognition  -  there  is  a  shift  away 
from  analytical  and  toward  intuitive  decision  making  processes  (Bois,  2002).  Thus,  a  key  defining 
characteristic  of  decision-making  expertise  is  recognizing  when  NOT  to  trust  your  intuition  and 
when  to  apply  more  rigorous  analytical  processes  (Bakken,  2008).  This  could  be  seen  as 
performing  a  higher  level  of  pattern  recognition  or  simply  identifying  a  pattern  (in  this  case  the 
pattern  “unmatched”  or  “unpredictable”). 

The  relationship  between  pattern  recognition  and  heuristics  is  still  under  debate  in  the  literature. 
Some  literature  suggests  that  heuristics  rely  on  pattern  recognition  in  order  to  identify  which 
heuristic  to  apply,  other  literature  suggests  that  heuristics  are  applied  when  no  pattern  emerges 
(Gonzales,  Lerch,  &  Lebiere,  2003).  All  this  builds  on  a  similar  idea  from  cue  integration:  that 
important  cognitive  processes  are  not  only  used  to  make  a  decision  but  may  also  be  used  to 
determine  which  decision  strategy  or  methods  to  apply  at  any  given  time.  This  is  iterated  by  Soh 
(2007)  as  well  as  Gigerenzer  (2008)  who  states  that  what  makes  a  good  decision  maker  is  the 
ability  to  recognize  the  right  strategy  for  the  situation. 

Pattern  Recognition-based  challenges 

As  with  cue  integration,  internal  and  environmental  variables  affect  pattern  recognition.  With 
satisficing,  for  example,  Gonzales,  Lerch,  and  Lebiere  (2003)  noted  that  the  level  of  satisfaction 
felt  by  the  decision  maker  considered  “enough”  will  depend  on  the  time  remaining  to  make  a 
decision.  This  implies  that  a  decision  maker’s  pattern  recognition  judgements  are  heavily 
dependent  on  time  constraints  (Kerstholt  &  Willems,  1993). 

Various  other  factors  also  affect  the  pattern  recognition  process.  Prospect  theory  shows  another 
example.  In  prospect  theory,  recognizing  an  outcome  as  a  gain  or  loss  will  influence  the  assessment 
of  risk,  which  will  in  turn  influence  the  decision  (Schulz,  1996).  Interestingly  risk  and  benefit  tend 
to  be  positively  correlated  in  reality,  but  people  perceive  them  as  negatively  correlated  (Slovic  et 
al.,  2005)  further  complicating  decision  makers  pattern  recognition  abilities. 

DDM  requires  decision  makers  to  create  mental  models,  think  critically  about  those  mental  models 
and  apply  them  to  novel  situations  (Cohen  et  al.,  1999).  Pattern  recognition  can  be  assumed  to  play 
a  role  in  mental  models,  both  in  their  identification  and  manipulation.  Thus  faulty  mental 
simulation  errors  which  are  commonly  made  by  decision  makers  in  recognition  primed  decision 
making  (Klein  &  Crandall,  1996)  could  also  be  seen  as  errors  in  pattern  recognition. 

Building  on  the  lack  of  use  of  feedback  by  decision  makers  in  DDM  scenarios  (see  Cue  Integration 
Section  above),  feedback  to  decision  makers  has  been  found  to  increase  the  error  rate.  Despite  this, 
subject  confidence  was  increased  as  people  made  faster  decisions  on  less  evidence  (Gough,  1992). 
Thus  DDM  decision  makers  are  not  incorporating  feedback  into  their  pattern  recognition  strategies. 
This  could  be  due  to  many  reasons,  perhaps  a  lack  of  understanding  of  the  implications  of  the 
feedback,  low  motivation  (Bois,  2002)  or  overconfidence.  More  exploration  of  this  is  required. 
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Training  pattern  recognition  in  a  DDM  context 

Over  learning,  to  the  point  of  automaticity,  is  important  in  training.  For  pattern  recognition,  this 
means  teaching  basic  skills  and  then  more  advanced  skills,  as  well  as  learning  the  importance  of 
mental  models  in  pattern  recognition  and  understanding  the  deeper  principles  which  make  patterns 
similar  (Thronesbery,  Sullivan,  Rhoads,  1996).  Effective  decision  makers  should  be  flexible,  quick, 
resilient,  adaptive,  risk  taking  and  accurate  (Canon-Bowers  &  Bell,  1997)  -  all  of  which  should 
improve  with  more  effective  pattern  recognition  abilities. 

Heuristics 

General  Themes 

Much  of  the  reason  decision  makers  use  heuristics  is  due  to  the  limited  amount  of  time,  cognitive 
resources,  and  information  constraints  within  the  environment  (Janser,  2007).  As  defined  earlier, 
DDM  contexts  are  highly  time  constrained  and  uncertain.  Heuristics  have  been  described  as  “short¬ 
cuts”  or  coping  strategies  as  a  method  of  dealing  with  time  pressures,  uncertainties,  and  large 
amounts  of  information  (Janser,  2007;  Rothrock,  &  Kirlik,  2003;  Todd  &  Gigerenzer,  1999).The 
ability  to  use  heuristics  to  cut  down  the  time  and  mental  resources  required  to  produce  a  decision  is 
not  a  surprise.  This  implies  that  within  a  DDM  context,  which  is  defined  by  these  factors,  heuristics 
are  inevitably  required. 

From  the  literature  reviewed,  several  contrasting  perspectives  were  identified.  They  are: 

•  Heuristics  are  negative  (nonoptimal)  (e.g.,  Stormer,  1991)  or  heuristics  are  useful  and 
accurate  decision-making  methods  (e.g.,  Soh,  2007;  Costello,  1992;  Goldenstein  & 
Gigerenzer,  2009);  and, 

•  Heuristics  are  avoidable  (e.g.,  Stormer,  1991;  Parchur  &  Hertwig,  2006)  or  heuristics  are 
unavoidable  due  to  either  human  cognitive  processing,  environmental  characteristics,  task 
characteristics,  or  a  combination  of  these  or  other  factors  (Butterfield,  1 993). 

Heuristic-based  challenges 

It  has  been  noted  in  a  number  of  studies  that  as  people  gain  experience,  there  is  a  shift  from 
analytical  towards  intuitive  decision  making  processes  (Bakken,  2008)  and  decision  making  is 
often  a  combination  of  both  (Dunwoody,  Haarbauer,  Mahan,  Marino,  &  Tang,  2000;  Payne, 
Bettman,  &  Johnson,  1992;  Rasmussen  &  Lind,  1982).  Errors  are  introduced  as  people  become 
more  reliant  on  intuitive  decision  making.  It  seems  people  are  not  able  to  accurately  assess  the 
limitations  of  their  own  intuitive  processes;  thus,  there  are  challenges  when  attempting  to 
understand  when  intuitive  processes  should  not  be  trusted  (Bakken,  2008). 

Another  challenge  is  the  high  amounts  of  cognitive  resources  and  experience  required  to  develop 
heuristics  (Cohen,  Thompson,  Adleman  &  Bresnick,  1999;  Rasmussen  &  Lind,  1982).  Gigerenzer 
and  Goldstein’s  idea  of  a  toolbox  of  fast,  frugal,  and  accurate  heuristics  supports  this  idea  that 
people  must  form  different  heuristics  for  different  decision  making  problems  (Martignon  & 
Hoffrage,  2002).  Challenges  in  using  heuristics  then  in  DDM  environments,  are  both  in  their 
development  and  the  implementation. 
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Training  and  heuristics  in  a  DDM  context 

The  disagreements  of  the  usefulness  and  avoidable  or  unavoidable  heuristics  significantly  impact 
the  direction  that  further  research  and  training  may  take.  With  the  idea  that  heuristics  are  negative 
and  avoidable,  one  may  attempt  to  educate  the  decision  maker  by  explicitly  identifying  detriments 
of  using  heuristics  (Butterfield,  1993;  Janser,  2007;  Stormer,  1991).  With  the  perspective  that 
heuristics  are  useful  and  unavoidable,  one  may  take  a  stance  to  monopolize  positive  effects.  For 
example,  Haynie  and  Shepherd  (2007)  provide  evidence  that  meta-cognitive  feedback  can  lead  to 
improved  decision  making.  In  this  instance,  meta-cognition  is  idealized  as  a  type  of  heuristic  that 
can  be  trained.  Rule-based  training  which  similarly  focuses  on  cognitive  aspects  was  also  shown  to 
result  in  more  robust  performance  in  dynamic  decision  making  (Kirklik,  Walker,  Fisk,  &  Nagel, 
1996).  Heuristic  training  based  on  this  perspective  may  alternatively  focus  on  critical  thinking 
strategies  (Cohen  et  al.,  1999)  and  rule-based  training  (Kirklik  et  al.,  1996;  Rasmussen  &  Link, 

1 982)  as  they  relate  to  contributing  to  positive  problem  solving  outcomes. 


Biases 

General  Themes 

Biases  can  become  evident  at  multiple  stages  within  the  decision  making  process  (Adams  et  al., 
2009).  Researchers  refer  to  biases  as  affecting  the  process  of  decision  making  by  mediating  the 
identification,  gathering,  and  interpretation  of  relevant  information  (Bakken,  2008;  Garcia-Retamero, 
Mueller,  Catena,  &  Maldonado,  2009;  Janser,  2007)  and  as  an  outcome  of  the  decision  making 
processes  (e.g.,  result  of  using  heuristics)  (Adelman  &  Bresnick,  1995;  Janser,  2007;  Rothrock  & 
Kirlik,  2003).  This  makes  general  findings  within  bias  literature  difficult  to  identify,  rather  general 
statements  may  be  best  summarised  for  particular  biases,  or  for  a  type  or  category  of  biases. 

Confusion  between  biases  and  heuristics  can  also  contribute  to  this  difficulty.  For  example, 
“assessment  of  risk”  has  been  reviewed  as  an  affect  heuristic  (Slovic,  Peters,  Funucane,  MacGregor, 
2005)  as  well  as  an  emotional-based  bias,  i.e.  “fads  and  fashion”  effect  (Schroeder,  2005). 

The  general  consensus  in  the  literature  seems  to  be  that  biases  are  associated  with  highly  uncertain 
and  time-constrained  decision  environments  (Butterfield,  1993).  Increased  situational  uncertainty 
and  time  pressure  requires  decision  makers  to  make  faster  decisions  which  are  more  prone  to  biases 
(Bakken,  2008).  This  suggests  that  the  number  of  biases  in  DDM  contexts  will  be  increased 
compared  to  more  simplistic  decision  making.  Other  themes  within  the  literature  demonstrate 
divided  perspectives  concerning  whether  biases  contribute  positively  or  negatively  to  decision 
making.  Rosenbloom,  Lee,  and  Unruh  (1993)  suggest  that  biases  can  help  organize  and  understand 
concepts,  and  reduce  computational  effort.  Others  argue  that  biases  can  only  have  negative  effects 
on  decision  making  (Brewer,  2005).  Even  for  those  who  do  support  that  biases  are  useful,  there  is 
high  agreement  that  biases  often  lead  to  faulty  decision  outcomes  (Rosenbloom,  Lee,  &  Unruh, 

1993).  It  has  been  generally  accepted  that  biases  are  most  often  applied  unconsciously  (Butterfield, 
1993;  Janser,  2007;  Stormer,  1991). 

Bias-based  challenges 

As  biases  may  be  incorporated  into  multiple  stages  within  the  decision  making  process,  cognitive 
challenges  may  result  as  the  effect  of  both  internal  and  external  influences.  Research  has  shown 
that  assigning  value  weightings  to  pieces  of  information  can  produce  biases  (Broder  &  Eichler, 
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2006;  Butterfield,  1993;  Gibb,  2000;  Schroeder,  2005).  It  has  also  been  proposed  that  when  values 
are  difficult  to  estimate,  those  values  may  be  ignored  entirely  (Payne,  Bettman,  &  Johnson,  1992), 
resulting  in  a  different  form  of  bias.  This  challenge  may  be  partly  due  to  cognitive  abilities,  but  also 
situational  factors  such  as  uncertainty. 

Influence  from  other  people  within  the  situation  can  also  dramatically  affect  biases.  For  example, 
the  attention  and  affect  within  a  context  can  change  people’s  willingness  to  take  risks.  The  “fads 
and  fashion”  effect  suggests  managers  are  more  likely  to  underestimate  risks  with  increased 
excitement  around  a  particular  industry  (Schroeder,  2005).  Challenges  presented  within  this  context 
may  be  more  social  in  nature  as  individual  biases  are  affected  by  not  only  the  situational 
information,  but  also  surrounding  relationships. 

In  DDM  contexts  it  is  assumed  that  information  overload  is  generally  high.  Attempting  to  prevent,  or 
analyze  biases  and  their  effects  on  decision  making  would  require  cognitive  resources  that  are  often 
unavailable.  Thus,  limitations  of  working  memory  produce  challenges  related  to  biases.  Also  within  a 
DDM  context,  decisions  are  interdependent.  Biases  that  affect  a  source  of  information  or  decision 
early  on  in  the  situation  may  result  in  confounding  errors  as  the  DDM  situation  progresses. 

Training  and  biases  in  a  DDM  context 

Continuing  from  the  idea  that  biases  are  unconsciously  applied,  several  researchers  advocate  that 
creating  awareness  about  biases,  that  is  to  educate  the  decision  maker  about  common  biases,  will  help 
to  minimize  their  negative  impact  in  decision  making  (Butterfield,  1993;  Janser,  2007;  Stormer, 

1991).  Thus  far  this  type  of  education  paradigm  has  not  been  shown  to  provide  consistent 
performance  increases,  thus  it  does  not  rid  decision  makers  of  their  biases  and  has  not  lead  to  any 
particular  type  of  training  methodology  for  changing  (increasing  or  decreasing)  biases  in  decision 
making  outcomes. 

Similar  to  the  discussion  about  heuristics,  it  may  be  important  to  identify  if  the  foundation  of 
training  will  be  the  presumption  that  biases  can  be  avoided  or  not.  Butterfield  (2005)  argues  that 
biases  (and  heuristics)  are  unavoidable  and  training  someone  to  keep  biases  out  of  decision  making 
is  futile.  Much  of  the  literature  recommends  focusing  training  on  analytic  type  processes  to  teach 
people  to  exercise  qualities  of  critical  judgement  (Butterfield,  2005;  Cohen,  Thompson,  Adelman, 

&  Bresnick,  1 999).  Other  methodologies  focus  on  improving  heuristics,  from  which  biases  are 
produced,  in  order  to  minimize  their  effects  (Brewer,  2005).  In  both  cases,  the  goal  is  to  rid  the 
decision  making  process  of  biases.  Those  who,  as  discussed  earlier,  are  of  the  perspective  that 
biases  can  be  helpful  may  focus  on  a  different  training  type  approach. 

Though  training  using  highly  analytical  processes  may  be  useful  in  transferring  DDM  skills,  we 
should  be  careful  not  to  apply  such  a  process  to  training  in  DDM  situations  as  DDM  contexts  may 
not  facilitate  the  time,  nor  environment  (e.g.,  certainty)  required.  For  this  reason,  some  suggestions 
of  overcoming  biases  may  not  be  appropriate  for  a  DDM  context.  For  example,  Stormer  (1991) 
suggests  that  the  decision  maker  should  take  more  time  to  reflect  on  COAs  being  considered  and 
pick  out  dominant  themes.  Unfortunately,  with  the  nature  of  DDM,  especially  the  time  constraints 
and  constantly  changing  information,  this  is  not  likely  to  be  a  useful  strategy. 
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Les  resultats  de  cette  etude  indiquaient  qu'il  y  avait  peu  de  documentation  sur  ('integration 
des  indices  dans  le  domaine  de  la  PDD.  Une  quantite  importante  de  documentation 
relative  aux  contextes  de  prise  de  decisions  plus  simples  a  done  ainsi  ete  utilisee  pour 
induire  le  role  de  ('integration  des  indices  dans  la  PDD.  On  estime  que  des  aspects 
particulars  de  ('integration  des  indices,  e'est-^-dire  la  perception,  la  memoire  de  travail,  la 
memoire  k  long  terme  et  la  retroaction,  jouent  un  role  important  dans  la  PDD.  Les 
implications  au  niveau  de  la  formation  sont  abord6es.  Des  recommandations  sont  faites 
sur  ce  qui  devrait  faire  I'objet  de  la  formation  et  la  fagon  de  former  les  gens.  Le  present 
document  se  termine  par  un  commentaire  de  conclusion  et  des  recommandations  sur  les 
recherches  a  venir. 
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